Jan 3 13:38:04 mds1 kernel: imklog 4.6.2, log source = /proc/kmsg started. Jan 3 13:38:04 mds1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1168" x-info="http://www.rsyslog.com"] (re)start Jan 3 13:44:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 13:49:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:49:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:49:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:49:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:49:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:49:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:50:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:50:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:50:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:50:35 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 13:50:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 13:50:35 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 13:50:35 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 13:50:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 48: Requesting the current CIB: S_POLICY_ENGINE Jan 3 13:50:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=48, ref=pe_calc-dc-1325616635-29, seq=4620, quorate=0 Jan 3 13:50:35 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 13:50:35 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 13:50:35 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 13:50:35 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 13:50:35 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 13:50:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 13:50:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 13:50:35 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 13:50:35 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 13:50:35 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 13:50:35 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 13:50:35 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 13:50:35 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 13:50:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 13:50:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 13:50:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 13:50:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 13:50:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 13:50:35 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 13:50:35 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 13:50:35 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 13:50:35 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 13:50:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 13:50:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 13:50:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 13:50:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 13:50:35 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 13:50:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 13:50:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 13:50:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 13:50:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 13:50:35 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 7: 0 actions in 0 synapses Jan 3 13:50:35 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 7 (ref=pe_calc-dc-1325616635-29) derived from /var/lib/pengine/pe-warn-46.bz2 Jan 3 13:50:35 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 13:50:35 mds1 crmd: [1305]: notice: run_graph: Transition 7 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-46.bz2): Complete Jan 3 13:50:35 mds1 crmd: [1305]: info: te_graph_trigger: Transition 7 is now complete Jan 3 13:50:35 mds1 crmd: [1305]: info: notify_crmd: Transition 7 status: done - Jan 3 13:50:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 13:50:35 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 13:50:35 mds1 pengine: [1304]: WARN: process_pe_message: Transition 7: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-46.bz2 Jan 3 13:50:35 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 13:50:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:51:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:51:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:51:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:51:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:52:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:52:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:52:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:52:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:52:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:53:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:53:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:53:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:53:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:53:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:54:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:54:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 13:54:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:54:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:55:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:55:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:55:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:55:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:55:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:56:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:56:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:56:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:56:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:57:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:57:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:57:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:58:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:58:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:58:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:58:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:58:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:59:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:59:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:59:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 13:59:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:00:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:00:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:00:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:00:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:01:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:01:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:01:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:01:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:02:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:02:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:02:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:02:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:02:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:03:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:03:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:03:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:03:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:03:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:03:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:04:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:04:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:04:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:04:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:04:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:05:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:05:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:05:35 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 14:05:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 14:05:35 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 14:05:35 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 14:05:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 49: Requesting the current CIB: S_POLICY_ENGINE Jan 3 14:05:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=49, ref=pe_calc-dc-1325617535-30, seq=4620, quorate=0 Jan 3 14:05:35 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 14:05:35 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 14:05:35 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 14:05:35 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 14:05:35 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 14:05:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 14:05:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 14:05:35 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:05:35 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:05:35 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:05:35 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:05:35 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 14:05:35 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 14:05:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 14:05:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 14:05:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 14:05:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 14:05:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 14:05:35 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 14:05:35 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 14:05:35 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 14:05:35 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 14:05:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 14:05:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 14:05:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 14:05:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 14:05:35 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 14:05:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 14:05:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 14:05:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 14:05:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 14:05:35 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 8: 0 actions in 0 synapses Jan 3 14:05:35 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 8 (ref=pe_calc-dc-1325617535-30) derived from /var/lib/pengine/pe-warn-47.bz2 Jan 3 14:05:35 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 14:05:35 mds1 crmd: [1305]: notice: run_graph: Transition 8 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-47.bz2): Complete Jan 3 14:05:35 mds1 crmd: [1305]: info: te_graph_trigger: Transition 8 is now complete Jan 3 14:05:35 mds1 crmd: [1305]: info: notify_crmd: Transition 8 status: done - Jan 3 14:05:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 14:05:35 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 14:05:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 8: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-47.bz2 Jan 3 14:05:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 14:05:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:06:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:06:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:06:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:06:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:06:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:07:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:07:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:07:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:07:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:08:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:08:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:08:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:08:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:09:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:09:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:09:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:09:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:09:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:09:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:10:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:10:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:10:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:10:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:11:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:11:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:11:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:11:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:11:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:12:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:12:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:12:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:13:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:13:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:13:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:13:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:13:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:13:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:14:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:14:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:14:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 3 14:14:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:14:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:15:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:15:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:15:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:15:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:16:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:16:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:16:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:16:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:16:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:17:04 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 14:17:04 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 14:17:04 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1387 seconds. Jan 3 14:20:35 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 14:20:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 14:20:35 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 14:20:35 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 14:20:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 50: Requesting the current CIB: S_POLICY_ENGINE Jan 3 14:20:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=50, ref=pe_calc-dc-1325618435-31, seq=4620, quorate=0 Jan 3 14:20:35 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 14:20:35 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 14:20:35 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 14:20:35 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 14:20:35 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 14:20:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 14:20:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 14:20:35 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:20:35 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:20:35 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:20:35 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:20:35 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 14:20:35 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 14:20:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 14:20:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 14:20:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 14:20:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 14:20:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 14:20:35 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 14:20:35 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 14:20:35 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 14:20:35 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 14:20:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 14:20:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 14:20:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 14:20:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 14:20:35 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 14:20:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 14:20:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 14:20:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 14:20:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 14:20:35 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 9: 0 actions in 0 synapses Jan 3 14:20:35 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 9 (ref=pe_calc-dc-1325618435-31) derived from /var/lib/pengine/pe-warn-48.bz2 Jan 3 14:20:35 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 14:20:35 mds1 crmd: [1305]: notice: run_graph: Transition 9 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-48.bz2): Complete Jan 3 14:20:35 mds1 crmd: [1305]: info: te_graph_trigger: Transition 9 is now complete Jan 3 14:20:35 mds1 crmd: [1305]: info: notify_crmd: Transition 9 status: done - Jan 3 14:20:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 14:20:35 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 14:20:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 9: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-48.bz2 Jan 3 14:20:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 14:24:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 3 14:35:35 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 14:35:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 14:35:35 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 14:35:35 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 14:35:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 51: Requesting the current CIB: S_POLICY_ENGINE Jan 3 14:35:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=51, ref=pe_calc-dc-1325619335-32, seq=4620, quorate=0 Jan 3 14:35:35 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 14:35:35 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 14:35:35 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 14:35:35 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 14:35:35 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 14:35:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 14:35:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 14:35:35 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:35:35 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:35:35 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:35:35 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:35:35 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 14:35:35 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 14:35:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 14:35:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 14:35:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 14:35:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 14:35:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 14:35:35 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 14:35:35 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 14:35:35 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 14:35:35 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 14:35:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 14:35:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 14:35:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 14:35:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 14:35:35 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 14:35:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 14:35:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 14:35:35 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 14:35:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 14:35:35 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 10: 0 actions in 0 synapses Jan 3 14:35:35 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 10 (ref=pe_calc-dc-1325619335-32) derived from /var/lib/pengine/pe-warn-49.bz2 Jan 3 14:35:35 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 14:35:35 mds1 crmd: [1305]: notice: run_graph: Transition 10 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-49.bz2): Complete Jan 3 14:35:35 mds1 crmd: [1305]: info: te_graph_trigger: Transition 10 is now complete Jan 3 14:35:35 mds1 crmd: [1305]: info: notify_crmd: Transition 10 status: done - Jan 3 14:35:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 14:35:35 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 14:35:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 10: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-49.bz2 Jan 3 14:35:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 14:40:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:40:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:40:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:40:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:40:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:41:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:41:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:41:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:41:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:41:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:42:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:42:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:42:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:42:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:43:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:43:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:43:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:43:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:44:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:44:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:44:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 14:44:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:44:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:45:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:45:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:45:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:45:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:46:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:46:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:46:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:46:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:47:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:47:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:47:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:48:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:48:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:48:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:48:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:48:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:48:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:49:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:49:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:49:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:49:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:50:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:50:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:50:35 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 14:50:35 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 14:50:35 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 14:50:35 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 14:50:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 52: Requesting the current CIB: S_POLICY_ENGINE Jan 3 14:50:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=52, ref=pe_calc-dc-1325620235-33, seq=4620, quorate=0 Jan 3 14:50:35 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 14:50:35 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 14:50:35 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 14:50:35 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 14:50:35 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 14:50:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 14:50:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 14:50:35 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:50:35 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:50:35 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:50:35 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 14:50:35 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 14:50:35 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 14:50:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 14:50:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 14:50:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 14:50:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 14:50:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 14:50:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 14:50:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 14:50:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 14:50:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 14:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 14:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 14:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 14:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 14:50:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 14:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 14:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 14:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 14:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 14:50:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 11: 0 actions in 0 synapses Jan 3 14:50:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 11 (ref=pe_calc-dc-1325620235-33) derived from /var/lib/pengine/pe-warn-50.bz2 Jan 3 14:50:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 14:50:36 mds1 crmd: [1305]: notice: run_graph: Transition 11 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-50.bz2): Complete Jan 3 14:50:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 11 is now complete Jan 3 14:50:36 mds1 crmd: [1305]: info: notify_crmd: Transition 11 status: done - Jan 3 14:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 14:50:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 14:50:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 11: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-50.bz2 Jan 3 14:50:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 14:50:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:51:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:51:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:51:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:51:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:51:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:52:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:52:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:52:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:52:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:53:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:53:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:53:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:53:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:53:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:54:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:54:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:54:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:54:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 3 14:54:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:54:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:55:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:55:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:55:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:55:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:56:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:56:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:56:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:56:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:56:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:56:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:57:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:57:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:57:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:57:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:57:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:58:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:58:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:58:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:58:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:58:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:58:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:59:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:59:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:59:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:59:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 14:59:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:00:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:00:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:00:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:00:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:01:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:01:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:01:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:01:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:02:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:02:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:02:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:03:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:03:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:03:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:03:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:03:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:04:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:04:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:04:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:04:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:05:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:05:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:05:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:05:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 15:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 15:05:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 15:05:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 15:05:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 53: Requesting the current CIB: S_POLICY_ENGINE Jan 3 15:05:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=53, ref=pe_calc-dc-1325621136-34, seq=4620, quorate=0 Jan 3 15:05:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 15:05:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 15:05:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 15:05:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 15:05:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 15:05:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 15:05:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 15:05:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:05:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:05:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:05:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:05:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 15:05:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 15:05:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 15:05:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 15:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 15:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 15:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 15:05:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 15:05:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 15:05:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 15:05:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 15:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 15:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 15:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 15:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 15:05:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 15:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 15:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 15:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 15:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 15:05:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 12: 0 actions in 0 synapses Jan 3 15:05:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 12 (ref=pe_calc-dc-1325621136-34) derived from /var/lib/pengine/pe-warn-51.bz2 Jan 3 15:05:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 15:05:36 mds1 crmd: [1305]: notice: run_graph: Transition 12 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-51.bz2): Complete Jan 3 15:05:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 12 is now complete Jan 3 15:05:36 mds1 crmd: [1305]: info: notify_crmd: Transition 12 status: done - Jan 3 15:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 15:05:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 15:05:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 12: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-51.bz2 Jan 3 15:05:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 15:05:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:05:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:05:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:06:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:06:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:06:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:06:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:07:06 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 15:07:06 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 15:07:06 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1566 seconds. Jan 3 15:14:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 3 15:20:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 15:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 15:20:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 15:20:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 15:20:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 54: Requesting the current CIB: S_POLICY_ENGINE Jan 3 15:20:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=54, ref=pe_calc-dc-1325622036-35, seq=4620, quorate=0 Jan 3 15:20:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 15:20:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 15:20:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 15:20:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 15:20:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 15:20:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 15:20:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 15:20:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:20:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:20:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:20:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:20:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 15:20:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 15:20:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 15:20:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 15:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 15:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 15:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 15:20:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 15:20:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 15:20:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 15:20:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 15:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 15:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 15:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 15:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 15:20:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 15:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 15:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 15:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 15:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 15:20:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 13: 0 actions in 0 synapses Jan 3 15:20:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 13 (ref=pe_calc-dc-1325622036-35) derived from /var/lib/pengine/pe-warn-52.bz2 Jan 3 15:20:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 15:20:36 mds1 crmd: [1305]: notice: run_graph: Transition 13 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-52.bz2): Complete Jan 3 15:20:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 13 is now complete Jan 3 15:20:36 mds1 crmd: [1305]: info: notify_crmd: Transition 13 status: done - Jan 3 15:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 15:20:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 15:20:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 13: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-52.bz2 Jan 3 15:20:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 15:24:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 3 15:33:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:33:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:33:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:33:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:33:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:33:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:34:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:34:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:34:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:34:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:35:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:35:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:35:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 15:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 15:35:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 15:35:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 15:35:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 55: Requesting the current CIB: S_POLICY_ENGINE Jan 3 15:35:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=55, ref=pe_calc-dc-1325622936-36, seq=4620, quorate=0 Jan 3 15:35:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 15:35:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 15:35:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 15:35:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 15:35:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 15:35:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 15:35:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 15:35:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:35:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:35:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:35:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:35:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 15:35:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 15:35:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 15:35:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 15:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 15:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 15:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 15:35:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 15:35:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 15:35:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 15:35:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 15:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 15:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 15:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 15:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 15:35:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 15:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 15:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 15:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 15:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 15:35:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 14: 0 actions in 0 synapses Jan 3 15:35:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 14 (ref=pe_calc-dc-1325622936-36) derived from /var/lib/pengine/pe-warn-53.bz2 Jan 3 15:35:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 15:35:36 mds1 crmd: [1305]: notice: run_graph: Transition 14 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-53.bz2): Complete Jan 3 15:35:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 14 is now complete Jan 3 15:35:36 mds1 crmd: [1305]: info: notify_crmd: Transition 14 status: done - Jan 3 15:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 15:35:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 15:35:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 14: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-53.bz2 Jan 3 15:35:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 15:35:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:36:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:36:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:36:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:36:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:37:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:37:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:37:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:37:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:37:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:38:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:38:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:38:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:38:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:39:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:39:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:39:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:39:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:39:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:40:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:40:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:40:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:40:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:41:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:41:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:41:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:41:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:41:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:42:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:42:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:42:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:43:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:43:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:43:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:43:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:44:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:44:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:44:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 15:44:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:44:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:44:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:45:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:45:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:45:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:45:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:46:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:46:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:46:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:46:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:47:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:47:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:47:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:47:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:47:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:48:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:48:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:48:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:48:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:49:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:49:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:49:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:50:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:50:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:50:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:50:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 15:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 15:50:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 15:50:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 15:50:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 56: Requesting the current CIB: S_POLICY_ENGINE Jan 3 15:50:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=56, ref=pe_calc-dc-1325623836-37, seq=4620, quorate=0 Jan 3 15:50:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 15:50:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 15:50:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 15:50:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 15:50:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 15:50:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 15:50:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 15:50:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:50:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:50:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:50:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 15:50:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 15:50:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 15:50:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 15:50:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 15:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 15:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 15:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 15:50:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 15:50:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 15:50:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 15:50:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 15:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 15:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 15:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 15:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 15:50:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 15:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 15:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 15:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 15:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 15:50:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 15: 0 actions in 0 synapses Jan 3 15:50:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 15 (ref=pe_calc-dc-1325623836-37) derived from /var/lib/pengine/pe-warn-54.bz2 Jan 3 15:50:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 15:50:36 mds1 crmd: [1305]: notice: run_graph: Transition 15 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-54.bz2): Complete Jan 3 15:50:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 15 is now complete Jan 3 15:50:36 mds1 crmd: [1305]: info: notify_crmd: Transition 15 status: done - Jan 3 15:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 15:50:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 15:50:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 15: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-54.bz2 Jan 3 15:50:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 15:50:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:50:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:51:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:51:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:51:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:51:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:52:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:52:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:52:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:52:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:53:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:53:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:53:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:53:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:53:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:54:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:54:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:54:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 15:54:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:54:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:55:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:55:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:55:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:55:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:55:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:56:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 15:56:12 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 15:56:12 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 15:56:12 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1286 seconds. Jan 3 16:05:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 16:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 16:05:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 16:05:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 16:05:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 57: Requesting the current CIB: S_POLICY_ENGINE Jan 3 16:05:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=57, ref=pe_calc-dc-1325624736-38, seq=4620, quorate=0 Jan 3 16:05:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 16:05:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 16:05:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 16:05:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 16:05:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 16:05:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 16:05:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 16:05:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:05:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:05:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:05:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:05:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 16:05:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 16:05:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 16:05:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 16:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 16:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 16:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 16:05:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 16:05:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 16:05:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 16:05:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 16:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 16:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 16:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 16:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 16:05:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 16:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 16:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 16:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 16:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 16:05:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 16: 0 actions in 0 synapses Jan 3 16:05:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 16 (ref=pe_calc-dc-1325624736-38) derived from /var/lib/pengine/pe-warn-55.bz2 Jan 3 16:05:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 16:05:36 mds1 crmd: [1305]: notice: run_graph: Transition 16 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-55.bz2): Complete Jan 3 16:05:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 16 is now complete Jan 3 16:05:36 mds1 crmd: [1305]: info: notify_crmd: Transition 16 status: done - Jan 3 16:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 16:05:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 16:05:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 16: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-55.bz2 Jan 3 16:05:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 16:14:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 16:17:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:17:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:17:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:17:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:18:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:18:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:18:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:19:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:19:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:19:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:19:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:19:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:20:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:20:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:20:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:20:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:20:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 16:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 16:20:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 16:20:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 16:20:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 58: Requesting the current CIB: S_POLICY_ENGINE Jan 3 16:20:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=58, ref=pe_calc-dc-1325625636-39, seq=4620, quorate=0 Jan 3 16:20:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 16:20:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 16:20:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 16:20:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 16:20:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 16:20:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 16:20:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 16:20:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:20:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:20:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:20:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:20:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 16:20:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 16:20:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 16:20:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 16:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 16:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 16:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 16:20:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 16:20:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 16:20:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 16:20:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 16:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 16:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 16:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 16:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 16:20:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 16:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 16:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 16:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 16:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 16:20:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 17: 0 actions in 0 synapses Jan 3 16:20:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 17 (ref=pe_calc-dc-1325625636-39) derived from /var/lib/pengine/pe-warn-56.bz2 Jan 3 16:20:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 16:20:36 mds1 crmd: [1305]: notice: run_graph: Transition 17 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-56.bz2): Complete Jan 3 16:20:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 17 is now complete Jan 3 16:20:36 mds1 crmd: [1305]: info: notify_crmd: Transition 17 status: done - Jan 3 16:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 16:20:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 16:20:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 17: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-56.bz2 Jan 3 16:20:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 16:20:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:20:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:21:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:21:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:21:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:21:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:22:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:22:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:22:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:22:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:22:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:23:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:23:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:23:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:23:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:23:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:23:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:24:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:24:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:24:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 3 16:24:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:24:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:24:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:25:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:25:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:25:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:26:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:26:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:26:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:26:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:26:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:27:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:27:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:27:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:27:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:27:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:27:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:28:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:28:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:28:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:28:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:29:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:29:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:29:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:29:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:29:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:30:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:30:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:30:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:30:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:30:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:31:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:31:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:31:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:31:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:31:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:32:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:32:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:32:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:32:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:32:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:33:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:33:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:33:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:33:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:33:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:34:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:34:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:34:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:34:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:35:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:35:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:35:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 16:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 16:35:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 16:35:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 16:35:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 59: Requesting the current CIB: S_POLICY_ENGINE Jan 3 16:35:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=59, ref=pe_calc-dc-1325626536-40, seq=4620, quorate=0 Jan 3 16:35:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 16:35:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 16:35:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 16:35:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 16:35:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 16:35:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 16:35:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 16:35:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:35:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:35:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:35:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:35:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 16:35:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 16:35:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 16:35:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 16:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 16:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 16:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 16:35:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 16:35:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 16:35:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 16:35:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 16:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 16:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 16:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 16:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 16:35:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 16:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 16:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 16:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 16:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 16:35:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 18: 0 actions in 0 synapses Jan 3 16:35:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 18 (ref=pe_calc-dc-1325626536-40) derived from /var/lib/pengine/pe-warn-57.bz2 Jan 3 16:35:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 16:35:36 mds1 crmd: [1305]: notice: run_graph: Transition 18 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-57.bz2): Complete Jan 3 16:35:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 18 is now complete Jan 3 16:35:36 mds1 crmd: [1305]: info: notify_crmd: Transition 18 status: done - Jan 3 16:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 16:35:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 16:35:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 18: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-57.bz2 Jan 3 16:35:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 16:35:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:35:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:36:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:36:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:36:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:36:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:36:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:37:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:37:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:37:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:37:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:37:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:38:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:38:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:38:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:38:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:39:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:39:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:39:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:39:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:39:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:40:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:40:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:40:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:40:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:41:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:41:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:41:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:41:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:42:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:42:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:42:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:42:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:43:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:43:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:43:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:43:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:43:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:44:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:44:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:44:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 16:44:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:44:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:44:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:45:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:45:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 16:45:36 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 16:45:36 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 16:45:36 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1521 seconds. Jan 3 16:50:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 16:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 16:50:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 16:50:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 16:50:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 60: Requesting the current CIB: S_POLICY_ENGINE Jan 3 16:50:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=60, ref=pe_calc-dc-1325627436-41, seq=4620, quorate=0 Jan 3 16:50:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 16:50:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 16:50:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 16:50:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 16:50:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 16:50:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 16:50:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 16:50:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:50:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:50:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:50:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 16:50:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 16:50:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 16:50:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 16:50:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 16:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 16:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 16:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 16:50:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 16:50:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 16:50:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 16:50:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 16:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 16:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 16:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 16:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 16:50:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 16:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 16:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 16:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 16:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 16:50:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 19: 0 actions in 0 synapses Jan 3 16:50:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 19 (ref=pe_calc-dc-1325627436-41) derived from /var/lib/pengine/pe-warn-58.bz2 Jan 3 16:50:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 16:50:36 mds1 crmd: [1305]: notice: run_graph: Transition 19 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-58.bz2): Complete Jan 3 16:50:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 19 is now complete Jan 3 16:50:36 mds1 crmd: [1305]: info: notify_crmd: Transition 19 status: done - Jan 3 16:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 16:50:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 16:50:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 19: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-58.bz2 Jan 3 16:50:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 16:54:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 17:05:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 17:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 17:05:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 17:05:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 17:05:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 61: Requesting the current CIB: S_POLICY_ENGINE Jan 3 17:05:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=61, ref=pe_calc-dc-1325628336-42, seq=4620, quorate=0 Jan 3 17:05:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 17:05:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 17:05:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 17:05:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 17:05:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 17:05:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 17:05:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 17:05:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:05:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:05:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:05:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:05:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 17:05:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 17:05:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 17:05:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 17:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 17:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 17:05:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 17:05:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 17:05:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 17:05:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 17:05:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 17:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 17:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 17:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 17:05:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 17:05:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 17:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 17:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 17:05:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 17:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 17:05:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 20: 0 actions in 0 synapses Jan 3 17:05:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 20 (ref=pe_calc-dc-1325628336-42) derived from /var/lib/pengine/pe-warn-59.bz2 Jan 3 17:05:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 17:05:36 mds1 crmd: [1305]: notice: run_graph: Transition 20 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-59.bz2): Complete Jan 3 17:05:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 20 is now complete Jan 3 17:05:36 mds1 crmd: [1305]: info: notify_crmd: Transition 20 status: done - Jan 3 17:05:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 17:05:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 17:05:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 20: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-59.bz2 Jan 3 17:05:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 17:10:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:11:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:11:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:11:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:11:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:11:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:12:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:12:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:12:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:12:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:12:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:13:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:13:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:13:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:14:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:14:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:14:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 3 17:14:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:14:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:15:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:15:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:15:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:15:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:16:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:16:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:16:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:17:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:17:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:17:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:17:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:17:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:18:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:18:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:18:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:18:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:19:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:19:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:19:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:19:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:20:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:20:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:20:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:20:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 17:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 17:20:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 17:20:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 17:20:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 62: Requesting the current CIB: S_POLICY_ENGINE Jan 3 17:20:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=62, ref=pe_calc-dc-1325629236-43, seq=4620, quorate=0 Jan 3 17:20:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 17:20:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 17:20:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 17:20:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 17:20:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 17:20:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 17:20:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 17:20:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:20:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:20:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:20:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:20:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 17:20:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 17:20:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 17:20:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 17:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 17:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 17:20:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 17:20:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 17:20:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 17:20:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 17:20:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 17:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 17:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 17:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 17:20:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 17:20:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 17:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 17:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 17:20:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 17:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 17:20:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 21: 0 actions in 0 synapses Jan 3 17:20:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 21 (ref=pe_calc-dc-1325629236-43) derived from /var/lib/pengine/pe-warn-60.bz2 Jan 3 17:20:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 17:20:36 mds1 crmd: [1305]: notice: run_graph: Transition 21 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-60.bz2): Complete Jan 3 17:20:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 21 is now complete Jan 3 17:20:36 mds1 crmd: [1305]: info: notify_crmd: Transition 21 status: done - Jan 3 17:20:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 17:20:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 17:20:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 21: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-60.bz2 Jan 3 17:20:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 17:20:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:21:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:21:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:21:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:21:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:22:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:22:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:22:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:23:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:23:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:23:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:23:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:24:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:24:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:24:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 17:24:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:24:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:24:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:25:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:25:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:25:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:25:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:26:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:26:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:26:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:26:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:26:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:27:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:27:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:27:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:28:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:28:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:28:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:28:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:29:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:29:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:29:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:29:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:30:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:30:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:30:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:30:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:30:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:31:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:31:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:31:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:31:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:31:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:32:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:32:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:32:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:32:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:33:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:33:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:33:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:33:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:33:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:34:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:34:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:34:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:34:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:34:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:35:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:35:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:35:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 17:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 17:35:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 17:35:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 17:35:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 63: Requesting the current CIB: S_POLICY_ENGINE Jan 3 17:35:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=63, ref=pe_calc-dc-1325630136-44, seq=4620, quorate=0 Jan 3 17:35:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 17:35:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 17:35:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 17:35:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 17:35:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 17:35:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 17:35:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 17:35:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:35:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:35:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:35:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:35:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 17:35:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 17:35:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 17:35:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 17:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 17:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 17:35:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 17:35:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 17:35:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 17:35:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 17:35:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 17:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 17:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 17:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 17:35:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 17:35:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 17:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 17:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 17:35:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 17:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 17:35:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 22: 0 actions in 0 synapses Jan 3 17:35:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 22 (ref=pe_calc-dc-1325630136-44) derived from /var/lib/pengine/pe-warn-61.bz2 Jan 3 17:35:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 17:35:36 mds1 crmd: [1305]: notice: run_graph: Transition 22 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-61.bz2): Complete Jan 3 17:35:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 22 is now complete Jan 3 17:35:36 mds1 crmd: [1305]: info: notify_crmd: Transition 22 status: done - Jan 3 17:35:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 17:35:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 17:35:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 22: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-61.bz2 Jan 3 17:35:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 17:35:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:35:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 17:36:10 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 17:36:10 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 17:36:10 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1518 seconds. Jan 3 17:44:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 17:50:36 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 17:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 17:50:36 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 17:50:36 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 17:50:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 64: Requesting the current CIB: S_POLICY_ENGINE Jan 3 17:50:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=64, ref=pe_calc-dc-1325631036-45, seq=4620, quorate=0 Jan 3 17:50:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 17:50:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 17:50:36 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 17:50:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 17:50:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 17:50:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 17:50:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 17:50:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:50:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:50:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:50:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 17:50:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 17:50:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 17:50:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 17:50:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 17:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 17:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 17:50:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 17:50:36 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 17:50:36 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 17:50:36 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 17:50:36 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 17:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 17:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 17:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 17:50:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 17:50:36 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 17:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 17:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 17:50:36 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 17:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 17:50:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 23: 0 actions in 0 synapses Jan 3 17:50:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 23 (ref=pe_calc-dc-1325631036-45) derived from /var/lib/pengine/pe-warn-62.bz2 Jan 3 17:50:36 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 17:50:36 mds1 crmd: [1305]: notice: run_graph: Transition 23 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-62.bz2): Complete Jan 3 17:50:36 mds1 crmd: [1305]: info: te_graph_trigger: Transition 23 is now complete Jan 3 17:50:36 mds1 crmd: [1305]: info: notify_crmd: Transition 23 status: done - Jan 3 17:50:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 17:50:36 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 17:50:36 mds1 pengine: [1304]: WARN: process_pe_message: Transition 23: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-62.bz2 Jan 3 17:50:36 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 17:54:30 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 18:13:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 18:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 18:13:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 18:13:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 18:13:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 65: Requesting the current CIB: S_POLICY_ENGINE Jan 3 18:13:55 mds1 corosync[1294]: [TOTEM ] A processor failed, forming new configuration. Jan 3 18:13:55 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4624: memb=1, new=0, lost=0 Jan 3 18:13:55 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 3 18:13:55 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4624: memb=1, new=0, lost=0 Jan 3 18:13:55 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 3 18:13:55 mds1 corosync[1294]: [pcmk ] info: update_member: 0xecb1c0 Node 1501210816 ((null)) born on: 4620 Jan 3 18:13:55 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 3 18:13:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=65, ref=pe_calc-dc-1325632435-46, seq=4620, quorate=0 Jan 3 18:13:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 18:13:55 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 3 18:13:55 mds1 corosync[1294]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 3 18:13:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 18:13:55 mds1 corosync[1294]: [MAIN ] Completed service synchronization, ready to provide service. Jan 3 18:13:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 18:13:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 18:13:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 18:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 18:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 18:13:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:13:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:13:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:13:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:13:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 18:13:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 18:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 18:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 18:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 18:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 18:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 18:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 18:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 18:13:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 18:13:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 18:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 18:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 18:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 18:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 18:13:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 18:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 18:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 18:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 18:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 18:13:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 24: 0 actions in 0 synapses Jan 3 18:13:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 24 (ref=pe_calc-dc-1325632435-46) derived from /var/lib/pengine/pe-warn-63.bz2 Jan 3 18:13:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 18:13:55 mds1 crmd: [1305]: notice: run_graph: Transition 24 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-63.bz2): Complete Jan 3 18:13:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 24 is now complete Jan 3 18:13:55 mds1 crmd: [1305]: info: notify_crmd: Transition 24 status: done - Jan 3 18:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 18:13:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 18:13:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:13:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 24: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-63.bz2 Jan 3 18:13:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 18:13:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:14:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:14:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:14:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:14:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:15:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:15:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:15:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:15:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:16:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:16:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:16:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:16:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:17:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:17:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:17:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:17:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:17:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:18:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:18:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:18:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:18:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:19:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:19:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:19:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:19:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:20:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:20:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:20:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:20:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:21:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:21:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:21:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:21:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:21:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:22:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:22:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:22:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:22:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:22:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:23:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:23:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:23:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:23:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 18:23:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:24:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:24:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:24:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:24:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:25:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:25:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:25:42 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 18:25:42 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 18:25:42 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1588 seconds. Jan 3 18:28:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 18:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 18:28:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 18:28:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 18:28:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 66: Requesting the current CIB: S_POLICY_ENGINE Jan 3 18:28:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=66, ref=pe_calc-dc-1325633335-47, seq=4620, quorate=0 Jan 3 18:28:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 18:28:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 18:28:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 18:28:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 18:28:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 18:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 18:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 18:28:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:28:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:28:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:28:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:28:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 18:28:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 18:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 18:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 18:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 18:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 18:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 18:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 18:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 18:28:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 18:28:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 18:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 18:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 18:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 18:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 18:28:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 18:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 18:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 18:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 18:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 18:28:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 25: 0 actions in 0 synapses Jan 3 18:28:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 25 (ref=pe_calc-dc-1325633335-47) derived from /var/lib/pengine/pe-warn-64.bz2 Jan 3 18:28:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 18:28:55 mds1 crmd: [1305]: notice: run_graph: Transition 25 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-64.bz2): Complete Jan 3 18:28:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 25 is now complete Jan 3 18:28:55 mds1 crmd: [1305]: info: notify_crmd: Transition 25 status: done - Jan 3 18:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 18:28:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 18:28:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 25: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-64.bz2 Jan 3 18:28:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 18:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 18:43:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 18:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 18:43:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 18:43:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 18:43:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 67: Requesting the current CIB: S_POLICY_ENGINE Jan 3 18:43:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=67, ref=pe_calc-dc-1325634235-48, seq=4620, quorate=0 Jan 3 18:43:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 18:43:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 18:43:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 18:43:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 18:43:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 18:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 18:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 18:43:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:43:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:43:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:43:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:43:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 18:43:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 18:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 18:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 18:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 18:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 18:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 18:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 18:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 18:43:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 18:43:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 18:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 18:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 18:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 18:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 18:43:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 18:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 18:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 18:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 18:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 18:43:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 26: 0 actions in 0 synapses Jan 3 18:43:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 26 (ref=pe_calc-dc-1325634235-48) derived from /var/lib/pengine/pe-warn-65.bz2 Jan 3 18:43:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 18:43:55 mds1 crmd: [1305]: notice: run_graph: Transition 26 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-65.bz2): Complete Jan 3 18:43:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 26 is now complete Jan 3 18:43:55 mds1 crmd: [1305]: info: notify_crmd: Transition 26 status: done - Jan 3 18:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 18:43:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 18:43:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 26: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-65.bz2 Jan 3 18:43:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 18:52:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:52:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:52:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:52:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:52:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:53:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:53:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:53:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:53:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 18:54:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:54:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:54:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:54:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:54:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:55:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:55:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:55:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:55:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:56:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:56:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:56:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:56:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:57:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:57:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:57:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:57:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:57:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:58:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:58:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:58:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:58:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:58:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:58:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 18:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 18:58:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 18:58:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 18:58:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 68: Requesting the current CIB: S_POLICY_ENGINE Jan 3 18:58:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=68, ref=pe_calc-dc-1325635135-49, seq=4620, quorate=0 Jan 3 18:58:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 18:58:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 18:58:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 18:58:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 18:58:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 18:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 18:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 18:58:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:58:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:58:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:58:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 18:58:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 18:58:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 18:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 18:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 18:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 18:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 18:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 18:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 18:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 18:58:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 18:58:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 18:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 18:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 18:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 18:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 18:58:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 18:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 18:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 18:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 18:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 18:58:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 27: 0 actions in 0 synapses Jan 3 18:58:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 27 (ref=pe_calc-dc-1325635135-49) derived from /var/lib/pengine/pe-warn-66.bz2 Jan 3 18:58:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 18:58:55 mds1 crmd: [1305]: notice: run_graph: Transition 27 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-66.bz2): Complete Jan 3 18:58:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 27 is now complete Jan 3 18:58:55 mds1 crmd: [1305]: info: notify_crmd: Transition 27 status: done - Jan 3 18:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 18:58:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 18:58:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 27: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-66.bz2 Jan 3 18:58:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 18:59:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:59:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:59:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 18:59:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:00:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:00:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:00:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:00:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:00:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:01:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:01:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:01:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:01:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:01:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:02:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:02:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:02:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:02:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:03:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:03:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:03:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:03:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 19:03:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:04:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:04:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:04:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:04:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:05:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:05:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:05:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:05:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:05:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:06:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:06:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:06:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:06:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:07:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:07:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:07:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:08:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:08:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:08:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:09:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:09:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:09:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:09:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:09:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:10:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:10:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:10:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:11:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:11:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:11:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:11:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:12:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:12:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:12:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:12:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:12:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:13:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:13:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:13:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:13:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:13:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 19:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 19:13:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 19:13:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 19:13:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 69: Requesting the current CIB: S_POLICY_ENGINE Jan 3 19:13:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=69, ref=pe_calc-dc-1325636035-50, seq=4620, quorate=0 Jan 3 19:13:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 19:13:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 19:13:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 19:13:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 19:13:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 19:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 19:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 19:13:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:13:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:13:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:13:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:13:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 19:13:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 19:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 19:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 19:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 19:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 19:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 19:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 19:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 19:13:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 19:13:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 19:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 19:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 19:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 19:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 19:13:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 19:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 19:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 19:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 19:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 19:13:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 28: 0 actions in 0 synapses Jan 3 19:13:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 28 (ref=pe_calc-dc-1325636035-50) derived from /var/lib/pengine/pe-warn-67.bz2 Jan 3 19:13:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 19:13:55 mds1 crmd: [1305]: notice: run_graph: Transition 28 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-67.bz2): Complete Jan 3 19:13:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 28 is now complete Jan 3 19:13:55 mds1 crmd: [1305]: info: notify_crmd: Transition 28 status: done - Jan 3 19:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 19:13:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 19:13:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 28: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-67.bz2 Jan 3 19:13:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 19:13:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:14:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:14:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:14:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:14:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:15:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:15:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:15:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:15:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:16:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:16:25 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 19:16:25 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 19:16:25 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1332 seconds. Jan 3 19:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 19:28:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 19:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 19:28:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 19:28:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 19:28:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 70: Requesting the current CIB: S_POLICY_ENGINE Jan 3 19:28:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=70, ref=pe_calc-dc-1325636935-51, seq=4620, quorate=0 Jan 3 19:28:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 19:28:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 19:28:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 19:28:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 19:28:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 19:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 19:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 19:28:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:28:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:28:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:28:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:28:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 19:28:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 19:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 19:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 19:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 19:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 19:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 19:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 19:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 19:28:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 19:28:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 19:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 19:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 19:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 19:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 19:28:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 19:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 19:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 19:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 19:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 19:28:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 29: 0 actions in 0 synapses Jan 3 19:28:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 29 (ref=pe_calc-dc-1325636935-51) derived from /var/lib/pengine/pe-warn-68.bz2 Jan 3 19:28:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 19:28:55 mds1 crmd: [1305]: notice: run_graph: Transition 29 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-68.bz2): Complete Jan 3 19:28:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 29 is now complete Jan 3 19:28:55 mds1 crmd: [1305]: info: notify_crmd: Transition 29 status: done - Jan 3 19:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 19:28:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 19:28:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 29: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-68.bz2 Jan 3 19:28:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 19:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 19:38:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:38:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:38:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:38:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:39:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:39:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:39:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:39:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:40:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:40:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:40:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:40:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:41:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:41:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:41:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:41:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:42:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:42:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:42:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:43:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:43:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:43:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:43:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:43:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 19:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 19:43:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 19:43:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 19:43:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 71: Requesting the current CIB: S_POLICY_ENGINE Jan 3 19:43:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=71, ref=pe_calc-dc-1325637835-52, seq=4620, quorate=0 Jan 3 19:43:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 19:43:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 19:43:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 19:43:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 19:43:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 19:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 19:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 19:43:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:43:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:43:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:43:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:43:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 19:43:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 19:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 19:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 19:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 19:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 19:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 19:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 19:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 19:43:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 19:43:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 19:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 19:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 19:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 19:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 19:43:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 19:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 19:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 19:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 19:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 19:43:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 30: 0 actions in 0 synapses Jan 3 19:43:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 30 (ref=pe_calc-dc-1325637835-52) derived from /var/lib/pengine/pe-warn-69.bz2 Jan 3 19:43:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 19:43:55 mds1 crmd: [1305]: notice: run_graph: Transition 30 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-69.bz2): Complete Jan 3 19:43:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 30 is now complete Jan 3 19:43:55 mds1 crmd: [1305]: info: notify_crmd: Transition 30 status: done - Jan 3 19:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 19:43:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 19:43:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 30: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-69.bz2 Jan 3 19:43:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 19:44:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:44:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:44:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:44:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:45:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:45:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:45:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:45:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:45:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:46:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:46:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:46:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:47:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:47:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:47:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:47:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:47:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:48:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:48:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:48:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:48:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:49:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:49:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:49:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:49:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:50:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:50:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:50:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:50:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:51:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:51:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:51:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:51:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:52:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:52:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:52:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:52:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:53:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:53:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:53:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:53:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 19:53:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:54:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:54:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:54:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:54:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:55:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:55:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:55:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:55:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:56:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:56:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:56:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:56:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:56:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:57:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:57:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:57:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:57:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:57:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:57:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:57:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:58:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:58:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:58:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:58:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:58:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 19:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 19:58:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 19:58:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 19:58:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 72: Requesting the current CIB: S_POLICY_ENGINE Jan 3 19:58:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=72, ref=pe_calc-dc-1325638735-53, seq=4620, quorate=0 Jan 3 19:58:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 19:58:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 19:58:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 19:58:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 19:58:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 19:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 19:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 19:58:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:58:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:58:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:58:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 19:58:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 19:58:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 19:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 19:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 19:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 19:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 19:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 19:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 19:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 19:58:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 19:58:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 19:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 19:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 19:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 19:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 19:58:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 19:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 19:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 19:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 19:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 19:58:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 31: 0 actions in 0 synapses Jan 3 19:58:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 31 (ref=pe_calc-dc-1325638735-53) derived from /var/lib/pengine/pe-warn-70.bz2 Jan 3 19:58:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 19:58:55 mds1 crmd: [1305]: notice: run_graph: Transition 31 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-70.bz2): Complete Jan 3 19:58:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 31 is now complete Jan 3 19:58:55 mds1 crmd: [1305]: info: notify_crmd: Transition 31 status: done - Jan 3 19:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 19:58:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 19:58:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 31: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-70.bz2 Jan 3 19:58:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 19:58:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:59:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:59:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:59:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 19:59:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:00:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:00:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:00:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:00:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:01:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:01:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:01:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:01:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:02:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:02:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:02:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:02:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:03:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:03:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:03:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:03:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:03:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 20:04:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:04:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:04:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:04:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:04:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:05:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:05:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:05:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:05:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:05:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:06:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:06:32 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 20:06:32 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 20:06:32 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1487 seconds. Jan 3 20:13:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 20:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 20:13:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 20:13:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 20:13:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 73: Requesting the current CIB: S_POLICY_ENGINE Jan 3 20:13:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=73, ref=pe_calc-dc-1325639635-54, seq=4620, quorate=0 Jan 3 20:13:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 20:13:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 20:13:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 20:13:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 20:13:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 20:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 20:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 20:13:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:13:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:13:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:13:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:13:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 20:13:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 20:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 20:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 20:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 20:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 20:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 20:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 20:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 20:13:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 20:13:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 20:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 20:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 20:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 20:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 20:13:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 20:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 20:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 20:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 20:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 20:13:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 32: 0 actions in 0 synapses Jan 3 20:13:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 32 (ref=pe_calc-dc-1325639635-54) derived from /var/lib/pengine/pe-warn-71.bz2 Jan 3 20:13:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 20:13:55 mds1 crmd: [1305]: notice: run_graph: Transition 32 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-71.bz2): Complete Jan 3 20:13:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 32 is now complete Jan 3 20:13:55 mds1 crmd: [1305]: info: notify_crmd: Transition 32 status: done - Jan 3 20:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 20:13:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 20:13:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 32: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-71.bz2 Jan 3 20:13:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 20:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 20:28:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 20:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 20:28:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 20:28:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 20:28:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 74: Requesting the current CIB: S_POLICY_ENGINE Jan 3 20:28:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=74, ref=pe_calc-dc-1325640535-55, seq=4620, quorate=0 Jan 3 20:28:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 20:28:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 20:28:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 20:28:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 20:28:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 20:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 20:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 20:28:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:28:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:28:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:28:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:28:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 20:28:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 20:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 20:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 20:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 20:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 20:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 20:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 20:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 20:28:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 20:28:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 20:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 20:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 20:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 20:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 20:28:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 20:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 20:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 20:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 20:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 20:28:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 33: 0 actions in 0 synapses Jan 3 20:28:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 33 (ref=pe_calc-dc-1325640535-55) derived from /var/lib/pengine/pe-warn-72.bz2 Jan 3 20:28:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 20:28:55 mds1 crmd: [1305]: notice: run_graph: Transition 33 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-72.bz2): Complete Jan 3 20:28:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 33 is now complete Jan 3 20:28:55 mds1 crmd: [1305]: info: notify_crmd: Transition 33 status: done - Jan 3 20:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 20:28:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 20:28:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 33: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-72.bz2 Jan 3 20:28:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 20:31:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:31:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:31:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:31:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:32:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:32:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:32:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:32:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:32:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:32:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:33:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:33:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:33:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:33:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 20:34:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:34:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:34:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:34:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:34:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:34:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:35:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:35:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:35:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:36:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:36:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:36:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:36:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:36:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:36:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:37:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:37:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:37:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:37:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:37:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:37:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:38:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:38:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:38:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:39:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:39:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:39:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:39:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:40:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:40:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:40:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:40:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:41:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:41:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:41:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:41:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:42:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:42:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:42:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:42:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:42:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:43:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:43:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:43:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:43:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 20:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 20:43:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 20:43:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 20:43:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 75: Requesting the current CIB: S_POLICY_ENGINE Jan 3 20:43:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=75, ref=pe_calc-dc-1325641435-56, seq=4620, quorate=0 Jan 3 20:43:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 20:43:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 20:43:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 20:43:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 20:43:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 20:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 20:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 20:43:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:43:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:43:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:43:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:43:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 20:43:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 20:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 20:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 20:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 20:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 20:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 20:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 20:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 20:43:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 20:43:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 20:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 20:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 20:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 20:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 20:43:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 20:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 20:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 20:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 20:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 20:43:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 34: 0 actions in 0 synapses Jan 3 20:43:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 34 (ref=pe_calc-dc-1325641435-56) derived from /var/lib/pengine/pe-warn-73.bz2 Jan 3 20:43:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 20:43:55 mds1 crmd: [1305]: notice: run_graph: Transition 34 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-73.bz2): Complete Jan 3 20:43:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 34 is now complete Jan 3 20:43:55 mds1 crmd: [1305]: info: notify_crmd: Transition 34 status: done - Jan 3 20:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 20:43:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 20:43:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 34: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-73.bz2 Jan 3 20:43:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 20:43:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:44:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:44:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:44:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:44:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:45:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:45:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:45:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:45:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:46:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:46:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:46:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:46:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:47:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:47:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:47:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:48:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:48:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:48:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:48:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:49:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:49:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:49:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:49:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:49:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:50:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:50:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:50:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:50:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:51:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:51:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:51:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:51:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:52:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:52:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:52:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:52:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:53:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:53:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:53:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:53:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 20:53:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:54:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:54:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:54:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:54:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:54:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:55:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:55:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:55:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:55:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:56:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:56:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:56:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:56:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:56:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 20:57:19 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 20:57:19 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 20:57:19 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1438 seconds. Jan 3 20:58:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 20:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 20:58:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 20:58:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 20:58:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 76: Requesting the current CIB: S_POLICY_ENGINE Jan 3 20:58:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=76, ref=pe_calc-dc-1325642335-57, seq=4620, quorate=0 Jan 3 20:58:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 20:58:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 20:58:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 20:58:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 20:58:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 20:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 20:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 20:58:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:58:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:58:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:58:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 20:58:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 20:58:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 20:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 20:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 20:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 20:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 20:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 20:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 20:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 20:58:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 20:58:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 20:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 20:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 20:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 20:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 20:58:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 20:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 20:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 20:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 20:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 20:58:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 35: 0 actions in 0 synapses Jan 3 20:58:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 35 (ref=pe_calc-dc-1325642335-57) derived from /var/lib/pengine/pe-warn-74.bz2 Jan 3 20:58:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 20:58:55 mds1 crmd: [1305]: notice: run_graph: Transition 35 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-74.bz2): Complete Jan 3 20:58:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 35 is now complete Jan 3 20:58:55 mds1 crmd: [1305]: info: notify_crmd: Transition 35 status: done - Jan 3 20:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 20:58:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 20:58:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 35: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-74.bz2 Jan 3 20:58:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 21:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 21:13:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 21:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 21:13:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 21:13:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 21:13:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 77: Requesting the current CIB: S_POLICY_ENGINE Jan 3 21:13:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=77, ref=pe_calc-dc-1325643235-58, seq=4620, quorate=0 Jan 3 21:13:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 21:13:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 21:13:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 21:13:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 21:13:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 21:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 21:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 21:13:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:13:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:13:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:13:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:13:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 21:13:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 21:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 21:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 21:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 21:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 21:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 21:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 21:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 21:13:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 21:13:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 21:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 21:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 21:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 21:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 21:13:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 21:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 21:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 21:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 21:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 21:13:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 36: 0 actions in 0 synapses Jan 3 21:13:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 36 (ref=pe_calc-dc-1325643235-58) derived from /var/lib/pengine/pe-warn-75.bz2 Jan 3 21:13:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 21:13:55 mds1 crmd: [1305]: notice: run_graph: Transition 36 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-75.bz2): Complete Jan 3 21:13:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 36 is now complete Jan 3 21:13:55 mds1 crmd: [1305]: info: notify_crmd: Transition 36 status: done - Jan 3 21:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 21:13:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 21:13:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 36: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-75.bz2 Jan 3 21:13:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 21:21:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:21:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:21:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:21:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:22:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:22:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:22:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:22:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:23:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:23:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:23:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:23:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 21:24:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:24:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:24:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:24:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:24:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:25:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:25:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:25:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:25:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:26:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:26:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:26:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:26:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:27:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:27:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:27:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:27:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:28:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:28:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:28:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:28:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:28:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 21:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 21:28:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 21:28:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 21:28:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 78: Requesting the current CIB: S_POLICY_ENGINE Jan 3 21:28:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=78, ref=pe_calc-dc-1325644135-59, seq=4620, quorate=0 Jan 3 21:28:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 21:28:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 21:28:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 21:28:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 21:28:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 21:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 21:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 21:28:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:28:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:28:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:28:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:28:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 21:28:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 21:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 21:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 21:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 21:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 21:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 21:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 21:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 21:28:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 21:28:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 21:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 21:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 21:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 21:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 21:28:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 21:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 21:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 21:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 21:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 21:28:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 37: 0 actions in 0 synapses Jan 3 21:28:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 37 (ref=pe_calc-dc-1325644135-59) derived from /var/lib/pengine/pe-warn-76.bz2 Jan 3 21:28:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 21:28:55 mds1 crmd: [1305]: notice: run_graph: Transition 37 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-76.bz2): Complete Jan 3 21:28:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 37 is now complete Jan 3 21:28:55 mds1 crmd: [1305]: info: notify_crmd: Transition 37 status: done - Jan 3 21:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 21:28:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 21:28:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 37: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-76.bz2 Jan 3 21:28:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 21:29:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:29:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:29:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:29:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:30:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:30:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:30:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:30:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:31:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:31:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:31:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:31:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:32:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:32:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:32:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:33:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:33:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:33:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:33:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:33:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 3 21:34:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:34:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:34:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:34:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:34:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:35:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:35:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:35:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:35:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:36:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:36:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:36:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:36:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:37:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:37:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:37:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:37:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:37:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:37:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:38:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:38:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:38:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:38:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:39:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:39:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:39:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:39:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:40:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:40:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:40:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:40:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:40:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:40:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:41:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:41:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:41:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:41:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:42:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:42:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:42:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:43:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:43:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:43:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:43:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 21:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 21:43:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 21:43:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 21:43:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 79: Requesting the current CIB: S_POLICY_ENGINE Jan 3 21:43:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=79, ref=pe_calc-dc-1325645035-60, seq=4620, quorate=0 Jan 3 21:43:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 21:43:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 21:43:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 21:43:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 21:43:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 21:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 21:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 21:43:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:43:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:43:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:43:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:43:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 21:43:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 21:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 21:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 21:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 21:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 21:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 21:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 21:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 21:43:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 21:43:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 21:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 21:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 21:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 21:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 21:43:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 21:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 21:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 21:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 21:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 21:43:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 38: 0 actions in 0 synapses Jan 3 21:43:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 38 (ref=pe_calc-dc-1325645035-60) derived from /var/lib/pengine/pe-warn-77.bz2 Jan 3 21:43:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 21:43:55 mds1 crmd: [1305]: notice: run_graph: Transition 38 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-77.bz2): Complete Jan 3 21:43:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 38 is now complete Jan 3 21:43:55 mds1 crmd: [1305]: info: notify_crmd: Transition 38 status: done - Jan 3 21:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 21:43:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 21:43:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 38: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-77.bz2 Jan 3 21:43:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 21:44:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:44:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:44:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:44:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:45:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:45:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:45:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:45:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:45:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:45:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:46:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:46:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:46:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:46:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:47:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:47:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:47:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:47:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 21:48:06 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 21:48:06 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 21:48:06 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1589 seconds. Jan 3 21:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 21:58:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 21:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 21:58:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 21:58:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 21:58:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 80: Requesting the current CIB: S_POLICY_ENGINE Jan 3 21:58:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=80, ref=pe_calc-dc-1325645935-61, seq=4620, quorate=0 Jan 3 21:58:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 21:58:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 21:58:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 21:58:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 21:58:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 21:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 21:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 21:58:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:58:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:58:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:58:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 21:58:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 21:58:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 21:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 21:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 21:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 21:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 21:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 21:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 21:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 21:58:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 21:58:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 21:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 21:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 21:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 21:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 21:58:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 21:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 21:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 21:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 21:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 21:58:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 39: 0 actions in 0 synapses Jan 3 21:58:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 39 (ref=pe_calc-dc-1325645935-61) derived from /var/lib/pengine/pe-warn-78.bz2 Jan 3 21:58:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 21:58:55 mds1 crmd: [1305]: notice: run_graph: Transition 39 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-78.bz2): Complete Jan 3 21:58:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 39 is now complete Jan 3 21:58:55 mds1 crmd: [1305]: info: notify_crmd: Transition 39 status: done - Jan 3 21:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 21:58:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 21:58:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 39: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-78.bz2 Jan 3 21:58:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 22:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 22:13:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 22:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 22:13:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 22:13:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 22:13:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 81: Requesting the current CIB: S_POLICY_ENGINE Jan 3 22:13:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=81, ref=pe_calc-dc-1325646835-62, seq=4620, quorate=0 Jan 3 22:13:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 22:13:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 22:13:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 22:13:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 22:13:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 22:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 22:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 22:13:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:13:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:13:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:13:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:13:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 22:13:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 22:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 22:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 22:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 22:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 22:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 22:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 22:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 22:13:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 22:13:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 22:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 22:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 22:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 22:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 22:13:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 22:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 22:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 22:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 22:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 22:13:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 40: 0 actions in 0 synapses Jan 3 22:13:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 40 (ref=pe_calc-dc-1325646835-62) derived from /var/lib/pengine/pe-warn-79.bz2 Jan 3 22:13:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 22:13:55 mds1 crmd: [1305]: notice: run_graph: Transition 40 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-79.bz2): Complete Jan 3 22:13:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 40 is now complete Jan 3 22:13:55 mds1 crmd: [1305]: info: notify_crmd: Transition 40 status: done - Jan 3 22:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 22:13:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 22:13:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 40: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-79.bz2 Jan 3 22:13:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 22:14:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:14:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:14:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:15:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:15:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:15:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:15:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:15:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:15:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:16:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:16:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:16:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:16:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:16:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:17:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:17:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:17:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:17:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:18:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:18:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:18:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:18:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:19:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:19:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:19:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:19:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:20:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:20:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:20:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:20:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:21:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:21:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:21:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:21:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:22:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:22:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:22:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:22:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:22:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:23:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:23:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:23:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:23:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:23:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 22:23:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:24:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:24:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:24:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:24:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:24:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:24:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:25:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:25:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:25:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:25:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:26:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:26:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:26:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:26:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:27:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:27:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:27:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:27:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:27:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:27:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:28:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:28:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:28:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:28:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:28:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 22:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 22:28:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 22:28:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 22:28:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 82: Requesting the current CIB: S_POLICY_ENGINE Jan 3 22:28:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=82, ref=pe_calc-dc-1325647735-63, seq=4620, quorate=0 Jan 3 22:28:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 22:28:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 22:28:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 22:28:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 22:28:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 22:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 22:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 22:28:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:28:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:28:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:28:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:28:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 22:28:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 22:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 22:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 22:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 22:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 22:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 22:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 22:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 22:28:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 22:28:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 22:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 22:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 22:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 22:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 22:28:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 22:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 22:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 22:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 22:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 22:28:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 41: 0 actions in 0 synapses Jan 3 22:28:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 41 (ref=pe_calc-dc-1325647735-63) derived from /var/lib/pengine/pe-warn-80.bz2 Jan 3 22:28:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 22:28:55 mds1 crmd: [1305]: notice: run_graph: Transition 41 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-80.bz2): Complete Jan 3 22:28:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 41 is now complete Jan 3 22:28:55 mds1 crmd: [1305]: info: notify_crmd: Transition 41 status: done - Jan 3 22:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 22:28:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 22:28:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 41: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-80.bz2 Jan 3 22:28:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 22:29:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:29:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:29:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:29:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:29:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:30:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:30:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:30:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:30:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:31:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:31:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:31:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:31:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:32:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:32:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:32:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:32:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:33:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:33:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:33:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:33:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 22:34:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:34:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:34:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:34:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:34:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:35:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:35:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:35:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:35:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:35:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:36:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:36:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:36:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:36:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:37:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:37:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:37:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:37:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:37:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 22:38:05 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 22:38:05 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 22:38:05 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1623 seconds. Jan 3 22:43:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 22:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 22:43:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 22:43:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 22:43:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 83: Requesting the current CIB: S_POLICY_ENGINE Jan 3 22:43:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=83, ref=pe_calc-dc-1325648635-64, seq=4620, quorate=0 Jan 3 22:43:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 22:43:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 22:43:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 22:43:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 22:43:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 22:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 22:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 22:43:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:43:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:43:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:43:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:43:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 22:43:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 22:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 22:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 22:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 22:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 22:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 22:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 22:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 22:43:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 22:43:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 22:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 22:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 22:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 22:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 22:43:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 22:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 22:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 22:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 22:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 22:43:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 42: 0 actions in 0 synapses Jan 3 22:43:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 42 (ref=pe_calc-dc-1325648635-64) derived from /var/lib/pengine/pe-warn-81.bz2 Jan 3 22:43:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 22:43:55 mds1 crmd: [1305]: notice: run_graph: Transition 42 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-81.bz2): Complete Jan 3 22:43:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 42 is now complete Jan 3 22:43:55 mds1 crmd: [1305]: info: notify_crmd: Transition 42 status: done - Jan 3 22:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 22:43:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 22:43:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 42: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-81.bz2 Jan 3 22:43:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 22:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 22:58:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 22:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 22:58:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 22:58:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 22:58:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 84: Requesting the current CIB: S_POLICY_ENGINE Jan 3 22:58:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=84, ref=pe_calc-dc-1325649535-65, seq=4620, quorate=0 Jan 3 22:58:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 22:58:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 22:58:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 22:58:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 22:58:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 22:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 22:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 22:58:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:58:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:58:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:58:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 22:58:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 22:58:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 22:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 22:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 22:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 22:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 22:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 22:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 22:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 22:58:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 22:58:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 22:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 22:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 22:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 22:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 22:58:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 22:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 22:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 22:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 22:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 22:58:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 43: 0 actions in 0 synapses Jan 3 22:58:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 43 (ref=pe_calc-dc-1325649535-65) derived from /var/lib/pengine/pe-warn-82.bz2 Jan 3 22:58:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 22:58:55 mds1 crmd: [1305]: notice: run_graph: Transition 43 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-82.bz2): Complete Jan 3 22:58:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 43 is now complete Jan 3 22:58:55 mds1 crmd: [1305]: info: notify_crmd: Transition 43 status: done - Jan 3 22:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 22:58:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 22:58:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 43: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-82.bz2 Jan 3 22:58:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 23:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 23:05:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:05:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:05:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:05:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:06:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:06:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:06:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:06:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:07:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:07:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:07:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:08:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:08:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:08:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:08:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:09:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:09:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:09:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:09:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:10:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:10:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:10:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:10:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:11:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:11:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:11:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:12:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:12:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:12:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:12:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:13:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:13:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:13:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 23:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 23:13:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 23:13:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 23:13:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 85: Requesting the current CIB: S_POLICY_ENGINE Jan 3 23:13:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=85, ref=pe_calc-dc-1325650435-66, seq=4620, quorate=0 Jan 3 23:13:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 23:13:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 23:13:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 23:13:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 23:13:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 23:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 23:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 23:13:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:13:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:13:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:13:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:13:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 23:13:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 23:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 23:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 23:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 23:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 23:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 23:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 23:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 23:13:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 23:13:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 23:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 23:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 23:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 23:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 23:13:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 23:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 23:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 23:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 23:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 23:13:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 44: 0 actions in 0 synapses Jan 3 23:13:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 44 (ref=pe_calc-dc-1325650435-66) derived from /var/lib/pengine/pe-warn-83.bz2 Jan 3 23:13:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 23:13:55 mds1 crmd: [1305]: notice: run_graph: Transition 44 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-83.bz2): Complete Jan 3 23:13:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 44 is now complete Jan 3 23:13:55 mds1 crmd: [1305]: info: notify_crmd: Transition 44 status: done - Jan 3 23:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 23:13:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 23:13:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 44: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-83.bz2 Jan 3 23:13:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 23:13:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:14:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:14:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:14:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:14:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:15:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:15:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:15:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:16:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:16:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:16:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:16:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:17:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:17:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:17:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:17:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:18:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:18:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:18:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:18:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:18:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:18:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:19:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:19:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:19:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:19:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:20:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:20:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:20:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:21:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:21:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:21:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:21:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:22:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:22:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:22:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:22:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:22:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:22:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:23:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:23:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:23:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:23:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 23:24:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:24:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:24:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:24:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:24:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:25:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:25:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:25:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:26:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:26:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:26:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:26:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:27:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:27:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:27:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:27:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:27:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:28:07 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 3 23:28:07 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 3 23:28:07 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1552 seconds. Jan 3 23:28:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 23:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 23:28:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 23:28:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 23:28:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 86: Requesting the current CIB: S_POLICY_ENGINE Jan 3 23:28:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=86, ref=pe_calc-dc-1325651335-67, seq=4620, quorate=0 Jan 3 23:28:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 23:28:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 23:28:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 23:28:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 23:28:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 23:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 23:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 23:28:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:28:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:28:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:28:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:28:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 23:28:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 23:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 23:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 23:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 23:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 23:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 23:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 23:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 23:28:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 23:28:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 23:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 23:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 23:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 23:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 23:28:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 23:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 23:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 23:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 23:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 23:28:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 45: 0 actions in 0 synapses Jan 3 23:28:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 45 (ref=pe_calc-dc-1325651335-67) derived from /var/lib/pengine/pe-warn-84.bz2 Jan 3 23:28:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 23:28:55 mds1 crmd: [1305]: notice: run_graph: Transition 45 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-84.bz2): Complete Jan 3 23:28:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 45 is now complete Jan 3 23:28:55 mds1 crmd: [1305]: info: notify_crmd: Transition 45 status: done - Jan 3 23:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 23:28:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 23:28:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 45: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-84.bz2 Jan 3 23:28:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 23:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 23:43:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 23:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 23:43:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 23:43:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 23:43:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 87: Requesting the current CIB: S_POLICY_ENGINE Jan 3 23:43:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=87, ref=pe_calc-dc-1325652235-68, seq=4620, quorate=0 Jan 3 23:43:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 23:43:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 23:43:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 23:43:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 23:43:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 23:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 23:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 23:43:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:43:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:43:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:43:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:43:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 23:43:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 23:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 23:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 23:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 23:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 23:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 23:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 23:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 23:43:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 23:43:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 23:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 23:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 23:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 23:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 23:43:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 23:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 23:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 23:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 23:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 23:43:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 46: 0 actions in 0 synapses Jan 3 23:43:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 46 (ref=pe_calc-dc-1325652235-68) derived from /var/lib/pengine/pe-warn-85.bz2 Jan 3 23:43:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 23:43:55 mds1 crmd: [1305]: notice: run_graph: Transition 46 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-85.bz2): Complete Jan 3 23:43:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 46 is now complete Jan 3 23:43:55 mds1 crmd: [1305]: info: notify_crmd: Transition 46 status: done - Jan 3 23:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 23:43:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 23:43:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 46: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-85.bz2 Jan 3 23:43:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 23:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 3 23:53:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:54:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:54:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:54:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:54:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:54:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:54:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:55:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:55:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:55:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:55:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:56:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:56:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:56:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:56:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:57:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:57:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:57:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:57:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:58:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:58:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:58:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:58:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:58:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 3 23:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 3 23:58:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 3 23:58:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 3 23:58:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 88: Requesting the current CIB: S_POLICY_ENGINE Jan 3 23:58:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=88, ref=pe_calc-dc-1325653135-69, seq=4620, quorate=0 Jan 3 23:58:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 3 23:58:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 3 23:58:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 3 23:58:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 3 23:58:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 3 23:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 3 23:58:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 3 23:58:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:58:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:58:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:58:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 3 23:58:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 3 23:58:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 3 23:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 3 23:58:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 3 23:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 3 23:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 3 23:58:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 3 23:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 3 23:58:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 3 23:58:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 3 23:58:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 3 23:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 3 23:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 3 23:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 3 23:58:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 3 23:58:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 3 23:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 3 23:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 3 23:58:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 3 23:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 3 23:58:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 47: 0 actions in 0 synapses Jan 3 23:58:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 47 (ref=pe_calc-dc-1325653135-69) derived from /var/lib/pengine/pe-warn-86.bz2 Jan 3 23:58:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 3 23:58:55 mds1 crmd: [1305]: notice: run_graph: Transition 47 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-86.bz2): Complete Jan 3 23:58:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 47 is now complete Jan 3 23:58:55 mds1 crmd: [1305]: info: notify_crmd: Transition 47 status: done - Jan 3 23:58:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 3 23:58:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 3 23:58:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 47: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-86.bz2 Jan 3 23:58:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 3 23:59:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:59:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:59:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 3 23:59:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:00:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:00:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:00:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:00:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:01:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:01:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:01:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:01:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:01:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:02:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:02:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:02:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:02:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:03:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:03:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:03:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:03:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:03:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (0.00us average, 0% utilization) in the last 10min Jan 4 00:04:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:04:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:04:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:04:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:05:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:05:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:05:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:05:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:05:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:06:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:06:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:06:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:06:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:06:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:07:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:07:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:07:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:08:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:08:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:08:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:08:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:09:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:09:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:09:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:09:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:09:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:10:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:10:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:10:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:10:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:10:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:11:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:11:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:11:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:11:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:11:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:12:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:12:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:12:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:12:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:13:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:13:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:13:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:13:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 00:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 00:13:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 00:13:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 00:13:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 89: Requesting the current CIB: S_POLICY_ENGINE Jan 4 00:13:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=89, ref=pe_calc-dc-1325654035-70, seq=4620, quorate=0 Jan 4 00:13:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 00:13:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 00:13:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 00:13:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 00:13:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 00:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 00:13:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 00:13:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:13:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:13:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:13:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:13:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 00:13:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 00:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 00:13:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 00:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 00:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 00:13:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 00:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 00:13:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 00:13:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 00:13:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 00:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 00:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 00:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 00:13:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 00:13:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 00:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 00:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 00:13:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 00:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 00:13:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 48: 0 actions in 0 synapses Jan 4 00:13:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 48 (ref=pe_calc-dc-1325654035-70) derived from /var/lib/pengine/pe-warn-87.bz2 Jan 4 00:13:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 00:13:55 mds1 crmd: [1305]: notice: run_graph: Transition 48 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-87.bz2): Complete Jan 4 00:13:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 48 is now complete Jan 4 00:13:55 mds1 crmd: [1305]: info: notify_crmd: Transition 48 status: done - Jan 4 00:13:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 00:13:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 00:13:55 mds1 pengine: [1304]: WARN: process_pe_message: Transition 48: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-87.bz2 Jan 4 00:13:55 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 00:14:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:14:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:14:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:14:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:15:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:15:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:15:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:15:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:15:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:16:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:16:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:16:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:16:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:17:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:17:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:17:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:17:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:18:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:18:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:18:39 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 00:18:39 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 00:18:39 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1570 seconds. Jan 4 00:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 00:28:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 00:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 00:28:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 00:28:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 00:28:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 90: Requesting the current CIB: S_POLICY_ENGINE Jan 4 00:28:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=90, ref=pe_calc-dc-1325654935-71, seq=4620, quorate=0 Jan 4 00:28:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 00:28:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 00:28:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 00:28:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 00:28:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 00:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 00:28:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 00:28:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:28:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:28:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:28:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:28:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 00:28:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 00:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 00:28:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 00:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 00:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 00:28:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 00:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 00:28:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 00:28:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 00:28:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 00:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 00:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 00:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 00:28:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 00:28:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 00:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 00:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 00:28:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 00:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 00:28:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 49: 0 actions in 0 synapses Jan 4 00:28:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 49 (ref=pe_calc-dc-1325654935-71) derived from /var/lib/pengine/pe-warn-88.bz2 Jan 4 00:28:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 00:28:55 mds1 crmd: [1305]: notice: run_graph: Transition 49 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-88.bz2): Complete Jan 4 00:28:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 49 is now complete Jan 4 00:28:55 mds1 crmd: [1305]: info: notify_crmd: Transition 49 status: done - Jan 4 00:28:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 00:28:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 00:28:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 49: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-88.bz2 Jan 4 00:28:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 00:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 4 00:43:55 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 00:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 00:43:55 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 00:43:55 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 00:43:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 91: Requesting the current CIB: S_POLICY_ENGINE Jan 4 00:43:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=91, ref=pe_calc-dc-1325655835-72, seq=4620, quorate=0 Jan 4 00:43:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 00:43:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 00:43:55 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 00:43:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 00:43:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 00:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 00:43:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 00:43:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:43:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:43:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:43:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:43:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 00:43:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 00:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 00:43:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 00:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 00:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 00:43:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 00:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 00:43:55 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 00:43:55 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 00:43:55 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 00:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 00:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 00:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 00:43:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 00:43:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 00:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 00:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 00:43:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 00:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 00:43:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 50: 0 actions in 0 synapses Jan 4 00:43:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 50 (ref=pe_calc-dc-1325655835-72) derived from /var/lib/pengine/pe-warn-89.bz2 Jan 4 00:43:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 00:43:55 mds1 crmd: [1305]: notice: run_graph: Transition 50 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-89.bz2): Complete Jan 4 00:43:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 50 is now complete Jan 4 00:43:55 mds1 crmd: [1305]: info: notify_crmd: Transition 50 status: done - Jan 4 00:43:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 00:43:55 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 00:43:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 50: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-89.bz2 Jan 4 00:43:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 00:44:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:44:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:45:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:45:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:45:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:45:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:46:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:46:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:46:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:46:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:46:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:46:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:47:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:47:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:47:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:47:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:47:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:47:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:48:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:48:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:48:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:48:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:49:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:49:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:49:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:50:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:50:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:50:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:50:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:50:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:51:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:51:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:51:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:51:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:52:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:52:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:52:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:52:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:53:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:53:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:53:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:53:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:53:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 00:54:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:54:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:54:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:54:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:54:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:55:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:55:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:55:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:55:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:56:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:56:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:56:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:56:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:57:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:57:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:57:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:57:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:58:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:58:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:58:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:58:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:58:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:58:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 00:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 00:58:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 00:58:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 00:58:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 92: Requesting the current CIB: S_POLICY_ENGINE Jan 4 00:58:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=92, ref=pe_calc-dc-1325656736-73, seq=4620, quorate=0 Jan 4 00:58:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 00:58:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 00:58:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 00:58:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 00:58:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 00:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 00:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 00:58:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:58:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:58:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:58:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 00:58:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 00:58:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 00:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 00:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 00:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 00:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 00:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 00:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 00:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 00:58:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 00:58:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 00:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 00:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 00:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 00:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 00:58:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 00:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 00:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 00:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 00:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 00:58:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 51: 0 actions in 0 synapses Jan 4 00:58:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 51 (ref=pe_calc-dc-1325656736-73) derived from /var/lib/pengine/pe-warn-90.bz2 Jan 4 00:58:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 00:58:56 mds1 crmd: [1305]: notice: run_graph: Transition 51 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-90.bz2): Complete Jan 4 00:58:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 51 is now complete Jan 4 00:58:56 mds1 crmd: [1305]: info: notify_crmd: Transition 51 status: done - Jan 4 00:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 00:58:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 00:58:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 51: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-90.bz2 Jan 4 00:58:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 00:58:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:59:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:59:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:59:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 00:59:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:00:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:00:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:00:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:00:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:00:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:01:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:01:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:01:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:01:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:02:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:02:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:02:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:02:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:02:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:03:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:03:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:03:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 4 01:03:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:04:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:04:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:04:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:04:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:05:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:05:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:05:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:05:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:05:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:06:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:06:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:06:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:06:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:06:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:07:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:07:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:07:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:07:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:07:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:08:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:08:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:08:44 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 01:08:44 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 01:08:44 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1536 seconds. Jan 4 01:13:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 01:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 01:13:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 01:13:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 01:13:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 93: Requesting the current CIB: S_POLICY_ENGINE Jan 4 01:13:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=93, ref=pe_calc-dc-1325657636-74, seq=4620, quorate=0 Jan 4 01:13:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 01:13:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 01:13:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 01:13:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 01:13:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 01:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 01:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 01:13:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:13:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:13:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:13:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:13:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 01:13:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 01:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 01:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 01:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 01:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 01:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 01:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 01:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 01:13:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 01:13:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 01:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 01:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 01:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 01:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 01:13:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 01:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 01:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 01:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 01:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 01:13:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 52: 0 actions in 0 synapses Jan 4 01:13:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 52 (ref=pe_calc-dc-1325657636-74) derived from /var/lib/pengine/pe-warn-91.bz2 Jan 4 01:13:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 01:13:56 mds1 crmd: [1305]: notice: run_graph: Transition 52 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-91.bz2): Complete Jan 4 01:13:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 52 is now complete Jan 4 01:13:56 mds1 crmd: [1305]: info: notify_crmd: Transition 52 status: done - Jan 4 01:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 01:13:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 01:13:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 52: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-91.bz2 Jan 4 01:13:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 01:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 01:28:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 01:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 01:28:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 01:28:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 01:28:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 94: Requesting the current CIB: S_POLICY_ENGINE Jan 4 01:28:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=94, ref=pe_calc-dc-1325658536-75, seq=4620, quorate=0 Jan 4 01:28:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 01:28:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 01:28:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 01:28:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 01:28:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 01:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 01:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 01:28:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:28:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:28:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:28:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:28:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 01:28:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 01:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 01:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 01:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 01:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 01:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 01:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 01:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 01:28:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 01:28:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 01:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 01:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 01:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 01:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 01:28:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 01:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 01:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 01:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 01:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 01:28:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 53: 0 actions in 0 synapses Jan 4 01:28:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 53 (ref=pe_calc-dc-1325658536-75) derived from /var/lib/pengine/pe-warn-92.bz2 Jan 4 01:28:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 01:28:56 mds1 crmd: [1305]: notice: run_graph: Transition 53 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-92.bz2): Complete Jan 4 01:28:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 53 is now complete Jan 4 01:28:56 mds1 crmd: [1305]: info: notify_crmd: Transition 53 status: done - Jan 4 01:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 01:28:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 01:28:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 53: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-92.bz2 Jan 4 01:28:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 01:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 01:34:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:34:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:34:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:34:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:35:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:35:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:35:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:35:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:35:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:36:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:36:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:36:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:36:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:37:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:37:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:37:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:37:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:37:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:38:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:38:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:38:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:38:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:38:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:39:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:39:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:39:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:39:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:40:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:40:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:40:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:40:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:41:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:41:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:41:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:41:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:42:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:42:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:42:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:43:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:43:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:43:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:43:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:43:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:43:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 01:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 01:43:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 01:43:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 01:43:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 95: Requesting the current CIB: S_POLICY_ENGINE Jan 4 01:43:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=95, ref=pe_calc-dc-1325659436-76, seq=4620, quorate=0 Jan 4 01:43:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 01:43:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 01:43:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 01:43:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 01:43:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 01:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 01:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 01:43:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:43:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:43:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:43:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:43:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 01:43:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 01:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 01:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 01:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 01:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 01:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 01:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 01:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 01:43:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 01:43:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 01:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 01:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 01:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 01:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 01:43:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 01:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 01:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 01:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 01:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 01:43:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 54: 0 actions in 0 synapses Jan 4 01:43:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 54 (ref=pe_calc-dc-1325659436-76) derived from /var/lib/pengine/pe-warn-93.bz2 Jan 4 01:43:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 01:43:56 mds1 crmd: [1305]: notice: run_graph: Transition 54 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-93.bz2): Complete Jan 4 01:43:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 54 is now complete Jan 4 01:43:56 mds1 crmd: [1305]: info: notify_crmd: Transition 54 status: done - Jan 4 01:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 01:43:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 01:43:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 54: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-93.bz2 Jan 4 01:43:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 01:44:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:44:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:44:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:44:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:45:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:45:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:45:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:45:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:45:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:46:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:46:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:46:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:46:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:47:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:47:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:47:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:47:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:48:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:48:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:48:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:49:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:49:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:49:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:49:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:49:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:50:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:50:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:50:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:51:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:51:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:51:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:51:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:51:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:52:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:52:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:52:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:52:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:53:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:53:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:53:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:53:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 4 01:54:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:54:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:54:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:54:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:54:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:55:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:55:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:55:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:56:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:56:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:56:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:56:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:57:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:57:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:57:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 01:57:37 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 01:57:37 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 01:57:37 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1621 seconds. Jan 4 01:58:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 01:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 01:58:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 01:58:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 01:58:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 96: Requesting the current CIB: S_POLICY_ENGINE Jan 4 01:58:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=96, ref=pe_calc-dc-1325660336-77, seq=4620, quorate=0 Jan 4 01:58:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 01:58:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 01:58:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 01:58:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 01:58:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 01:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 01:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 01:58:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:58:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:58:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:58:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 01:58:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 01:58:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 01:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 01:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 01:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 01:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 01:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 01:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 01:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 01:58:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 01:58:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 01:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 01:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 01:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 01:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 01:58:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 01:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 01:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 01:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 01:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 01:58:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 55: 0 actions in 0 synapses Jan 4 01:58:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 55 (ref=pe_calc-dc-1325660336-77) derived from /var/lib/pengine/pe-warn-94.bz2 Jan 4 01:58:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 01:58:56 mds1 crmd: [1305]: notice: run_graph: Transition 55 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-94.bz2): Complete Jan 4 01:58:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 55 is now complete Jan 4 01:58:56 mds1 crmd: [1305]: info: notify_crmd: Transition 55 status: done - Jan 4 01:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 01:58:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 01:58:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 55: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-94.bz2 Jan 4 01:58:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 02:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 02:13:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 02:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 02:13:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 02:13:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 02:13:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 97: Requesting the current CIB: S_POLICY_ENGINE Jan 4 02:13:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=97, ref=pe_calc-dc-1325661236-78, seq=4620, quorate=0 Jan 4 02:13:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 02:13:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 02:13:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 02:13:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 02:13:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 02:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 02:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 02:13:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:13:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:13:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:13:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:13:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 02:13:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 02:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 02:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 02:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 02:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 02:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 02:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 02:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 02:13:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 02:13:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 02:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 02:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 02:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 02:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 02:13:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 02:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 02:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 02:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 02:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 02:13:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 56: 0 actions in 0 synapses Jan 4 02:13:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 56 (ref=pe_calc-dc-1325661236-78) derived from /var/lib/pengine/pe-warn-95.bz2 Jan 4 02:13:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 02:13:56 mds1 crmd: [1305]: notice: run_graph: Transition 56 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-95.bz2): Complete Jan 4 02:13:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 56 is now complete Jan 4 02:13:56 mds1 crmd: [1305]: info: notify_crmd: Transition 56 status: done - Jan 4 02:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 02:13:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 02:13:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 56: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-95.bz2 Jan 4 02:13:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 02:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 02:24:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:24:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:24:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:25:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:25:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:25:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:25:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:25:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:26:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:26:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:26:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:26:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:26:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:27:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:27:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:27:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:27:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:27:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:28:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:28:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:28:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:28:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:28:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:28:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 02:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 02:28:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 02:28:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 02:28:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 98: Requesting the current CIB: S_POLICY_ENGINE Jan 4 02:28:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=98, ref=pe_calc-dc-1325662136-79, seq=4620, quorate=0 Jan 4 02:28:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 02:28:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 02:28:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 02:28:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 02:28:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 02:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 02:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 02:28:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:28:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:28:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:28:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:28:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 02:28:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 02:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 02:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 02:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 02:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 02:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 02:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 02:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 02:28:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 02:28:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 02:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 02:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 02:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 02:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 02:28:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 02:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 02:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 02:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 02:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 02:28:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 57: 0 actions in 0 synapses Jan 4 02:28:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 57 (ref=pe_calc-dc-1325662136-79) derived from /var/lib/pengine/pe-warn-96.bz2 Jan 4 02:28:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 02:28:56 mds1 crmd: [1305]: notice: run_graph: Transition 57 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-96.bz2): Complete Jan 4 02:28:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 57 is now complete Jan 4 02:28:56 mds1 crmd: [1305]: info: notify_crmd: Transition 57 status: done - Jan 4 02:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 02:28:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 02:28:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 57: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-96.bz2 Jan 4 02:28:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 02:29:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:29:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:29:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:29:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:30:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:30:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:30:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:30:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:31:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:31:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:31:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:31:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:31:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:32:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:32:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:32:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:32:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:33:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:33:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:33:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:33:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:33:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 4 02:33:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:34:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:34:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:34:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:34:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:34:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:34:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:35:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:35:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:35:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:35:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:35:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:36:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:36:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:36:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:36:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:36:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:37:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:37:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:37:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:37:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:38:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:38:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:38:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:38:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:39:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:39:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:39:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:39:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:40:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:40:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:40:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:40:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:40:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:41:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:41:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:41:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:42:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:42:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:42:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:42:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:42:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:43:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:43:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:43:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:43:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:43:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:43:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 02:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 02:43:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 02:43:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 02:43:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 99: Requesting the current CIB: S_POLICY_ENGINE Jan 4 02:43:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=99, ref=pe_calc-dc-1325663036-80, seq=4620, quorate=0 Jan 4 02:43:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 02:43:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 02:43:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 02:43:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 02:43:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 02:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 02:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 02:43:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:43:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:43:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:43:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:43:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 02:43:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 02:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 02:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 02:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 02:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 02:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 02:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 02:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 02:43:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 02:43:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 02:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 02:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 02:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 02:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 02:43:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 02:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 02:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 02:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 02:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 02:43:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 58: 0 actions in 0 synapses Jan 4 02:43:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 58 (ref=pe_calc-dc-1325663036-80) derived from /var/lib/pengine/pe-warn-97.bz2 Jan 4 02:43:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 02:43:56 mds1 crmd: [1305]: notice: run_graph: Transition 58 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-97.bz2): Complete Jan 4 02:43:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 58 is now complete Jan 4 02:43:56 mds1 crmd: [1305]: info: notify_crmd: Transition 58 status: done - Jan 4 02:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 02:43:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 02:43:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 58: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-97.bz2 Jan 4 02:43:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 02:44:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:44:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:44:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:44:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:44:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:45:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:45:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:45:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:45:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:45:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:45:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:46:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:46:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:46:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:46:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:47:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:47:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 02:47:41 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 02:47:41 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 02:47:41 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1445 seconds. Jan 4 02:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 02:58:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 02:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 02:58:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 02:58:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 02:58:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 100: Requesting the current CIB: S_POLICY_ENGINE Jan 4 02:58:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=100, ref=pe_calc-dc-1325663936-81, seq=4620, quorate=0 Jan 4 02:58:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 02:58:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 02:58:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 02:58:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 02:58:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 02:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 02:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 02:58:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:58:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:58:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:58:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 02:58:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 02:58:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 02:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 02:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 02:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 02:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 02:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 02:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 02:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 02:58:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 02:58:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 02:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 02:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 02:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 02:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 02:58:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 02:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 02:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 02:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 02:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 02:58:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 59: 0 actions in 0 synapses Jan 4 02:58:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 59 (ref=pe_calc-dc-1325663936-81) derived from /var/lib/pengine/pe-warn-98.bz2 Jan 4 02:58:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 02:58:56 mds1 crmd: [1305]: notice: run_graph: Transition 59 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-98.bz2): Complete Jan 4 02:58:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 59 is now complete Jan 4 02:58:56 mds1 crmd: [1305]: info: notify_crmd: Transition 59 status: done - Jan 4 02:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 02:58:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 02:58:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 59: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-98.bz2 Jan 4 02:58:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 03:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 03:11:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:11:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:12:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:12:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:12:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:12:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:12:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:12:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:13:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:13:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:13:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:13:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:13:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:13:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:13:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 03:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 03:13:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 03:13:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 03:13:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 101: Requesting the current CIB: S_POLICY_ENGINE Jan 4 03:13:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=101, ref=pe_calc-dc-1325664836-82, seq=4620, quorate=0 Jan 4 03:13:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 03:13:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 03:13:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 03:13:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 03:13:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 03:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 03:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 03:13:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:13:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:13:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:13:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:13:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 03:13:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 03:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 03:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 03:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 03:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 03:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 03:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 03:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 03:13:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 03:13:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 03:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 03:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 03:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 03:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 03:13:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 03:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 03:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 03:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 03:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 03:13:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 60: 0 actions in 0 synapses Jan 4 03:13:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 60 (ref=pe_calc-dc-1325664836-82) derived from /var/lib/pengine/pe-warn-99.bz2 Jan 4 03:13:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 03:13:56 mds1 crmd: [1305]: notice: run_graph: Transition 60 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-99.bz2): Complete Jan 4 03:13:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 60 is now complete Jan 4 03:13:56 mds1 crmd: [1305]: info: notify_crmd: Transition 60 status: done - Jan 4 03:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 03:13:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 03:13:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 60: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-99.bz2 Jan 4 03:13:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 03:14:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:14:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:14:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:14:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:14:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:15:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:15:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:15:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:15:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:15:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:16:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:16:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:16:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:16:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:17:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:17:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:17:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:17:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:18:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:18:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:18:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:18:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:18:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:19:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:19:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:19:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:19:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:20:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:20:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:20:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:21:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:21:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:21:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:21:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:21:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:22:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:22:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:22:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:22:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:22:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:23:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:23:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:23:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:23:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:23:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 4 03:24:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:24:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:24:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:24:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:25:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:25:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:25:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:26:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:26:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:26:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:26:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:27:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:27:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:27:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:27:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:28:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:28:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:28:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:28:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:28:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 03:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 03:28:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 03:28:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 03:28:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 102: Requesting the current CIB: S_POLICY_ENGINE Jan 4 03:28:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=102, ref=pe_calc-dc-1325665736-83, seq=4620, quorate=0 Jan 4 03:28:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 03:28:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 03:28:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 03:28:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 03:28:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 03:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 03:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 03:28:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:28:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:28:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:28:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:28:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 03:28:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 03:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 03:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 03:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 03:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 03:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 03:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 03:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 03:28:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 03:28:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 03:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 03:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 03:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 03:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 03:28:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 03:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 03:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 03:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 03:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 03:28:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 61: 0 actions in 0 synapses Jan 4 03:28:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 61 (ref=pe_calc-dc-1325665736-83) derived from /var/lib/pengine/pe-warn-100.bz2 Jan 4 03:28:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 03:28:56 mds1 crmd: [1305]: notice: run_graph: Transition 61 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-100.bz2): Complete Jan 4 03:28:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 61 is now complete Jan 4 03:28:56 mds1 crmd: [1305]: info: notify_crmd: Transition 61 status: done - Jan 4 03:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 03:28:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 03:28:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 61: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-100.bz2 Jan 4 03:28:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 03:28:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:29:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:29:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:29:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:29:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:30:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:30:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:30:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:30:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:31:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:31:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:31:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:31:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:32:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:32:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:32:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:32:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:32:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:33:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:33:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:33:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:33:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 03:34:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:34:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:34:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:34:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:34:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:35:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:35:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:35:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:36:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:36:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:36:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:36:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:36:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:37:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:37:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:37:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:37:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:37:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 03:38:12 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 03:38:12 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 03:38:12 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1323 seconds. Jan 4 03:43:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 03:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 03:43:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 03:43:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 03:43:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 103: Requesting the current CIB: S_POLICY_ENGINE Jan 4 03:43:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=103, ref=pe_calc-dc-1325666636-84, seq=4620, quorate=0 Jan 4 03:43:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 03:43:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 03:43:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 03:43:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 03:43:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 03:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 03:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 03:43:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:43:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:43:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:43:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:43:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 03:43:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 03:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 03:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 03:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 03:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 03:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 03:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 03:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 03:43:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 03:43:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 03:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 03:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 03:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 03:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 03:43:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 03:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 03:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 03:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 03:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 03:43:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 62: 0 actions in 0 synapses Jan 4 03:43:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 62 (ref=pe_calc-dc-1325666636-84) derived from /var/lib/pengine/pe-warn-101.bz2 Jan 4 03:43:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 03:43:56 mds1 crmd: [1305]: notice: run_graph: Transition 62 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-101.bz2): Complete Jan 4 03:43:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 62 is now complete Jan 4 03:43:56 mds1 crmd: [1305]: info: notify_crmd: Transition 62 status: done - Jan 4 03:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 03:43:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 03:43:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 62: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-101.bz2 Jan 4 03:43:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 03:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 03:58:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 03:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 03:58:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 03:58:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 03:58:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 104: Requesting the current CIB: S_POLICY_ENGINE Jan 4 03:58:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=104, ref=pe_calc-dc-1325667536-85, seq=4620, quorate=0 Jan 4 03:58:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 03:58:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 03:58:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 03:58:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 03:58:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 03:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 03:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 03:58:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:58:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:58:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:58:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 03:58:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 03:58:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 03:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 03:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 03:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 03:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 03:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 03:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 03:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 03:58:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 03:58:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 03:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 03:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 03:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 03:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 03:58:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 03:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 03:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 03:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 03:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 03:58:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 63: 0 actions in 0 synapses Jan 4 03:58:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 63 (ref=pe_calc-dc-1325667536-85) derived from /var/lib/pengine/pe-warn-102.bz2 Jan 4 03:58:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 03:58:56 mds1 crmd: [1305]: notice: run_graph: Transition 63 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-102.bz2): Complete Jan 4 03:58:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 63 is now complete Jan 4 03:58:56 mds1 crmd: [1305]: info: notify_crmd: Transition 63 status: done - Jan 4 03:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 03:58:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 03:58:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 63: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-102.bz2 Jan 4 03:58:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 04:00:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:00:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:00:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:00:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:01:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:01:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:01:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:01:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:02:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:02:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:02:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:02:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:02:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:02:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:03:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:03:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:03:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:03:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:03:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 04:04:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:04:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:04:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:04:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:04:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:05:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:05:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:05:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:05:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:05:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:06:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:06:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:06:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:07:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:07:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:07:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:07:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:07:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:07:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:07:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:08:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:08:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:08:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:08:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:09:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:09:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:09:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:09:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:09:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:10:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:10:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:10:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:10:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:11:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:11:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:11:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:11:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:11:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:12:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:12:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:12:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:12:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:13:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:13:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:13:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:13:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:13:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:13:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 04:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 04:13:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 04:13:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 04:13:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 105: Requesting the current CIB: S_POLICY_ENGINE Jan 4 04:13:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=105, ref=pe_calc-dc-1325668436-86, seq=4620, quorate=0 Jan 4 04:13:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 04:13:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 04:13:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 04:13:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 04:13:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 04:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 04:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 04:13:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:13:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:13:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:13:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:13:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 04:13:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 04:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 04:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 04:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 04:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 04:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 04:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 04:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 04:13:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 04:13:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 04:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 04:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 04:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 04:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 04:13:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 04:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 04:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 04:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 04:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 04:13:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 64: 0 actions in 0 synapses Jan 4 04:13:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 64 (ref=pe_calc-dc-1325668436-86) derived from /var/lib/pengine/pe-warn-103.bz2 Jan 4 04:13:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 04:13:56 mds1 crmd: [1305]: notice: run_graph: Transition 64 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-103.bz2): Complete Jan 4 04:13:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 64 is now complete Jan 4 04:13:56 mds1 crmd: [1305]: info: notify_crmd: Transition 64 status: done - Jan 4 04:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 04:13:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 04:13:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 64: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-103.bz2 Jan 4 04:13:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 04:14:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:14:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:14:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:14:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:14:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:15:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:15:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:15:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:15:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:16:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:16:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:16:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:17:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:17:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:17:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:17:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:18:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:18:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:18:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:18:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:19:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:19:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:19:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:19:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:20:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:20:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:20:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:20:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:21:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:21:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:21:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:21:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:21:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:22:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:22:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:22:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:22:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:23:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:23:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:23:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 04:23:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:24:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:24:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:24:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:24:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:25:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:25:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:25:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:25:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:25:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:26:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:26:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:26:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:26:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:26:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:27:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:27:39 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 04:27:39 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 04:27:39 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1487 seconds. Jan 4 04:28:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 04:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 04:28:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 04:28:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 04:28:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 106: Requesting the current CIB: S_POLICY_ENGINE Jan 4 04:28:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=106, ref=pe_calc-dc-1325669336-87, seq=4620, quorate=0 Jan 4 04:28:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 04:28:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 04:28:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 04:28:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 04:28:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 04:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 04:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 04:28:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:28:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:28:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:28:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:28:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 04:28:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 04:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 04:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 04:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 04:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 04:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 04:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 04:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 04:28:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 04:28:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 04:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 04:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 04:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 04:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 04:28:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 04:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 04:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 04:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 04:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 04:28:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 65: 0 actions in 0 synapses Jan 4 04:28:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 65 (ref=pe_calc-dc-1325669336-87) derived from /var/lib/pengine/pe-warn-104.bz2 Jan 4 04:28:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 04:28:56 mds1 crmd: [1305]: notice: run_graph: Transition 65 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-104.bz2): Complete Jan 4 04:28:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 65 is now complete Jan 4 04:28:56 mds1 crmd: [1305]: info: notify_crmd: Transition 65 status: done - Jan 4 04:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 04:28:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 04:28:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 65: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-104.bz2 Jan 4 04:28:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 04:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 04:43:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 04:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 04:43:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 04:43:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 04:43:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 107: Requesting the current CIB: S_POLICY_ENGINE Jan 4 04:43:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=107, ref=pe_calc-dc-1325670236-88, seq=4620, quorate=0 Jan 4 04:43:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 04:43:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 04:43:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 04:43:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 04:43:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 04:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 04:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 04:43:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:43:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:43:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:43:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:43:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 04:43:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 04:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 04:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 04:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 04:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 04:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 04:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 04:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 04:43:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 04:43:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 04:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 04:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 04:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 04:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 04:43:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 04:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 04:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 04:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 04:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 04:43:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 66: 0 actions in 0 synapses Jan 4 04:43:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 66 (ref=pe_calc-dc-1325670236-88) derived from /var/lib/pengine/pe-warn-105.bz2 Jan 4 04:43:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 04:43:56 mds1 crmd: [1305]: notice: run_graph: Transition 66 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-105.bz2): Complete Jan 4 04:43:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 66 is now complete Jan 4 04:43:56 mds1 crmd: [1305]: info: notify_crmd: Transition 66 status: done - Jan 4 04:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 04:43:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 04:43:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 66: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-105.bz2 Jan 4 04:43:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 04:52:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:52:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:52:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:52:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:53:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:53:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:53:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:53:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 04:54:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:54:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:54:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:54:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:55:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:55:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:55:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:55:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:55:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:56:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:56:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:56:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:56:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:57:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:57:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:57:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:57:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:57:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:57:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:58:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:58:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:58:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:58:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:58:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 04:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 04:58:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 04:58:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 04:58:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 108: Requesting the current CIB: S_POLICY_ENGINE Jan 4 04:58:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=108, ref=pe_calc-dc-1325671136-89, seq=4620, quorate=0 Jan 4 04:58:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 04:58:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 04:58:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 04:58:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 04:58:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 04:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 04:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 04:58:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:58:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:58:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:58:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 04:58:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 04:58:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 04:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 04:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 04:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 04:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 04:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 04:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 04:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 04:58:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 04:58:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 04:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 04:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 04:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 04:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 04:58:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 04:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 04:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 04:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 04:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 04:58:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 67: 0 actions in 0 synapses Jan 4 04:58:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 67 (ref=pe_calc-dc-1325671136-89) derived from /var/lib/pengine/pe-warn-106.bz2 Jan 4 04:58:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 04:58:56 mds1 crmd: [1305]: notice: run_graph: Transition 67 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-106.bz2): Complete Jan 4 04:58:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 67 is now complete Jan 4 04:58:56 mds1 crmd: [1305]: info: notify_crmd: Transition 67 status: done - Jan 4 04:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 04:58:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 04:58:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 67: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-106.bz2 Jan 4 04:58:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 04:59:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:59:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:59:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:59:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 04:59:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:00:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:00:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:00:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:00:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:00:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:01:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:01:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:01:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:01:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:01:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:02:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:02:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:02:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:02:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:03:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:03:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:03:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 05:03:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:04:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:04:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:04:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:04:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:05:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:05:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:05:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:05:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:05:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:06:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:06:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:06:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:06:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:07:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:07:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:07:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:07:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:08:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:08:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:08:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:08:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:08:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:09:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:09:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:09:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:09:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:10:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:10:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:10:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:10:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:10:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:11:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:11:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:11:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:11:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:11:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:11:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:12:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:12:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:12:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:12:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:13:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:13:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:13:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:13:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 05:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 05:13:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 05:13:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 05:13:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 109: Requesting the current CIB: S_POLICY_ENGINE Jan 4 05:13:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=109, ref=pe_calc-dc-1325672036-90, seq=4620, quorate=0 Jan 4 05:13:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 05:13:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 05:13:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 05:13:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 05:13:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 05:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 05:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 05:13:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:13:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:13:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:13:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:13:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 05:13:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 05:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 05:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 05:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 05:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 05:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 05:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 05:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 05:13:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 05:13:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 05:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 05:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 05:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 05:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 05:13:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 05:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 05:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 05:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 05:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 05:13:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 68: 0 actions in 0 synapses Jan 4 05:13:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 68 (ref=pe_calc-dc-1325672036-90) derived from /var/lib/pengine/pe-warn-107.bz2 Jan 4 05:13:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 05:13:56 mds1 crmd: [1305]: notice: run_graph: Transition 68 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-107.bz2): Complete Jan 4 05:13:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 68 is now complete Jan 4 05:13:56 mds1 crmd: [1305]: info: notify_crmd: Transition 68 status: done - Jan 4 05:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 05:13:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 05:13:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 68: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-107.bz2 Jan 4 05:13:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 05:14:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:14:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:14:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:14:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:15:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:15:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:15:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:15:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:16:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:16:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:16:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:16:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:16:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:17:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:17:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:17:21 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 05:17:21 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 05:17:21 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1626 seconds. Jan 4 05:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 05:28:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 05:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 05:28:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 05:28:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 05:28:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 110: Requesting the current CIB: S_POLICY_ENGINE Jan 4 05:28:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=110, ref=pe_calc-dc-1325672936-91, seq=4620, quorate=0 Jan 4 05:28:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 05:28:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 05:28:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 05:28:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 05:28:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 05:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 05:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 05:28:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:28:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:28:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:28:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:28:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 05:28:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 05:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 05:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 05:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 05:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 05:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 05:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 05:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 05:28:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 05:28:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 05:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 05:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 05:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 05:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 05:28:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 05:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 05:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 05:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 05:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 05:28:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 69: 0 actions in 0 synapses Jan 4 05:28:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 69 (ref=pe_calc-dc-1325672936-91) derived from /var/lib/pengine/pe-warn-108.bz2 Jan 4 05:28:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 05:28:56 mds1 crmd: [1305]: notice: run_graph: Transition 69 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-108.bz2): Complete Jan 4 05:28:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 69 is now complete Jan 4 05:28:56 mds1 crmd: [1305]: info: notify_crmd: Transition 69 status: done - Jan 4 05:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 05:28:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 05:28:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 69: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-108.bz2 Jan 4 05:28:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 05:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 05:43:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 05:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 05:43:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 05:43:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 05:43:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 111: Requesting the current CIB: S_POLICY_ENGINE Jan 4 05:43:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=111, ref=pe_calc-dc-1325673836-92, seq=4620, quorate=0 Jan 4 05:43:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 05:43:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 05:43:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 05:43:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 05:43:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 05:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 05:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 05:43:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:43:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:43:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:43:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:43:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 05:43:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 05:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 05:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 05:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 05:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 05:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 05:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 05:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 05:43:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 05:43:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 05:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 05:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 05:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 05:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 05:43:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 05:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 05:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 05:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 05:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 05:43:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 70: 0 actions in 0 synapses Jan 4 05:43:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 70 (ref=pe_calc-dc-1325673836-92) derived from /var/lib/pengine/pe-warn-109.bz2 Jan 4 05:43:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 05:43:56 mds1 crmd: [1305]: notice: run_graph: Transition 70 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-109.bz2): Complete Jan 4 05:43:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 70 is now complete Jan 4 05:43:56 mds1 crmd: [1305]: info: notify_crmd: Transition 70 status: done - Jan 4 05:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 05:43:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 05:43:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 70: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-109.bz2 Jan 4 05:43:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 05:44:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:44:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:44:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:45:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:45:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:45:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:45:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:45:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:45:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:46:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:46:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:46:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:46:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:46:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:47:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:47:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:47:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:48:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:48:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:48:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:48:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:48:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:48:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:49:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:49:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:49:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:49:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:49:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:50:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:50:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:50:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:50:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:50:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:51:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:51:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:51:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:51:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:52:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:52:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:52:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:52:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:52:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:52:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:53:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:53:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:53:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:53:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 05:53:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:54:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:54:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:54:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:55:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:55:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:55:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:55:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:55:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:56:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:56:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:56:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:56:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:57:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:57:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:57:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:57:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:58:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:58:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:58:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:58:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:58:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:58:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:58:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 05:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 05:58:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 05:58:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 05:58:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 112: Requesting the current CIB: S_POLICY_ENGINE Jan 4 05:58:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=112, ref=pe_calc-dc-1325674736-93, seq=4620, quorate=0 Jan 4 05:58:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 05:58:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 05:58:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 05:58:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 05:58:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 05:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 05:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 05:58:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:58:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:58:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:58:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 05:58:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 05:58:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 05:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 05:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 05:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 05:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 05:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 05:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 05:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 05:58:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 05:58:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 05:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 05:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 05:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 05:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 05:58:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 05:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 05:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 05:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 05:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 05:58:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 71: 0 actions in 0 synapses Jan 4 05:58:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 71 (ref=pe_calc-dc-1325674736-93) derived from /var/lib/pengine/pe-warn-110.bz2 Jan 4 05:58:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 05:58:56 mds1 crmd: [1305]: notice: run_graph: Transition 71 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-110.bz2): Complete Jan 4 05:58:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 71 is now complete Jan 4 05:58:56 mds1 crmd: [1305]: info: notify_crmd: Transition 71 status: done - Jan 4 05:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 05:58:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 05:58:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 71: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-110.bz2 Jan 4 05:58:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 05:59:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:59:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:59:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:59:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 05:59:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:00:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:00:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:00:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:00:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:00:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:01:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:01:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:01:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:01:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:02:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:02:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:02:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:02:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:03:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:03:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:03:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:03:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 06:04:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:04:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:04:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:04:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:05:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:05:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:05:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:05:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:05:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:06:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:06:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:06:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:06:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:07:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:07:11 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 06:07:11 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 06:07:11 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1483 seconds. Jan 4 06:13:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 06:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 06:13:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 06:13:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 06:13:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 113: Requesting the current CIB: S_POLICY_ENGINE Jan 4 06:13:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=113, ref=pe_calc-dc-1325675636-94, seq=4620, quorate=0 Jan 4 06:13:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 06:13:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 06:13:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 06:13:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 06:13:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 06:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 06:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 06:13:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:13:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:13:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:13:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:13:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 06:13:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 06:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 06:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 06:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 06:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 06:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 06:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 06:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 06:13:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 06:13:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 06:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 06:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 06:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 06:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 06:13:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 06:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 06:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 06:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 06:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 06:13:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 72: 0 actions in 0 synapses Jan 4 06:13:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 72 (ref=pe_calc-dc-1325675636-94) derived from /var/lib/pengine/pe-warn-111.bz2 Jan 4 06:13:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 06:13:56 mds1 crmd: [1305]: notice: run_graph: Transition 72 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-111.bz2): Complete Jan 4 06:13:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 72 is now complete Jan 4 06:13:56 mds1 crmd: [1305]: info: notify_crmd: Transition 72 status: done - Jan 4 06:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 06:13:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 06:13:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 72: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-111.bz2 Jan 4 06:13:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 06:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 06:28:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 06:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 06:28:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 06:28:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 06:28:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 114: Requesting the current CIB: S_POLICY_ENGINE Jan 4 06:28:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=114, ref=pe_calc-dc-1325676536-95, seq=4620, quorate=0 Jan 4 06:28:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 06:28:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 06:28:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 06:28:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 06:28:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 06:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 06:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 06:28:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:28:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:28:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:28:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:28:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 06:28:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 06:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 06:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 06:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 06:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 06:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 06:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 06:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 06:28:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 06:28:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 06:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 06:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 06:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 06:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 06:28:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 06:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 06:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 06:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 06:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 06:28:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 73: 0 actions in 0 synapses Jan 4 06:28:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 73 (ref=pe_calc-dc-1325676536-95) derived from /var/lib/pengine/pe-warn-112.bz2 Jan 4 06:28:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 06:28:56 mds1 crmd: [1305]: notice: run_graph: Transition 73 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-112.bz2): Complete Jan 4 06:28:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 73 is now complete Jan 4 06:28:56 mds1 crmd: [1305]: info: notify_crmd: Transition 73 status: done - Jan 4 06:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 06:28:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 06:28:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 73: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-112.bz2 Jan 4 06:28:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 06:31:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:32:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:32:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:32:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:32:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:33:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:33:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:33:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:33:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (20000.00us average, 0% utilization) in the last 10min Jan 4 06:34:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:34:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:34:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:34:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:35:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:35:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:35:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:35:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:36:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:36:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:36:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:36:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:37:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:37:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:37:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:38:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:38:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:38:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:38:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:39:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:39:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:39:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:39:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:39:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:40:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:40:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:40:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:40:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:40:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:40:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:41:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:41:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:41:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:41:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:41:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:42:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:42:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:42:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:42:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:42:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:43:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:43:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:43:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:43:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:43:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:43:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 06:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 06:43:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 06:43:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 06:43:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 115: Requesting the current CIB: S_POLICY_ENGINE Jan 4 06:43:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=115, ref=pe_calc-dc-1325677436-96, seq=4620, quorate=0 Jan 4 06:43:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 06:43:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 06:43:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 06:43:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 06:43:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 06:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 06:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 06:43:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:43:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:43:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:43:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:43:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 06:43:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 06:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 06:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 06:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 06:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 06:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 06:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 06:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 06:43:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 06:43:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 06:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 06:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 06:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 06:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 06:43:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 06:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 06:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 06:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 06:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 06:43:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 74: 0 actions in 0 synapses Jan 4 06:43:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 74 (ref=pe_calc-dc-1325677436-96) derived from /var/lib/pengine/pe-warn-113.bz2 Jan 4 06:43:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 06:43:56 mds1 crmd: [1305]: notice: run_graph: Transition 74 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-113.bz2): Complete Jan 4 06:43:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 74 is now complete Jan 4 06:43:56 mds1 crmd: [1305]: info: notify_crmd: Transition 74 status: done - Jan 4 06:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 06:43:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 06:43:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 74: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-113.bz2 Jan 4 06:43:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 06:44:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:44:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:44:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:44:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:44:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:45:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:45:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:45:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:45:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:45:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:46:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:46:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:46:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:46:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:46:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:47:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:47:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:47:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:47:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:48:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:48:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:48:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:48:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:48:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:49:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:49:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:49:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:50:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:50:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:50:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:50:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:50:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:51:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:51:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:51:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:51:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:52:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:52:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:52:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:52:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:53:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:53:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:53:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:53:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 06:54:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:54:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:54:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:54:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:54:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:55:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:55:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:55:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:55:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:56:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:56:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:56:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:56:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:56:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 06:57:06 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 06:57:06 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 06:57:06 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1249 seconds. Jan 4 06:58:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 06:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 06:58:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 06:58:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 06:58:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 116: Requesting the current CIB: S_POLICY_ENGINE Jan 4 06:58:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=116, ref=pe_calc-dc-1325678336-97, seq=4620, quorate=0 Jan 4 06:58:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 06:58:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 06:58:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 06:58:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 06:58:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 06:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 06:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 06:58:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:58:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:58:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:58:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 06:58:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 06:58:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 06:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 06:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 06:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 06:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 06:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 06:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 06:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 06:58:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 06:58:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 06:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 06:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 06:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 06:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 06:58:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 06:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 06:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 06:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 06:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 06:58:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 75: 0 actions in 0 synapses Jan 4 06:58:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 75 (ref=pe_calc-dc-1325678336-97) derived from /var/lib/pengine/pe-warn-114.bz2 Jan 4 06:58:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 06:58:56 mds1 crmd: [1305]: notice: run_graph: Transition 75 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-114.bz2): Complete Jan 4 06:58:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 75 is now complete Jan 4 06:58:56 mds1 crmd: [1305]: info: notify_crmd: Transition 75 status: done - Jan 4 06:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 06:58:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 06:58:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 75: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-114.bz2 Jan 4 06:58:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 07:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 07:13:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 07:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 07:13:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 07:13:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 07:13:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 117: Requesting the current CIB: S_POLICY_ENGINE Jan 4 07:13:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=117, ref=pe_calc-dc-1325679236-98, seq=4620, quorate=0 Jan 4 07:13:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 07:13:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 07:13:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 07:13:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 07:13:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 07:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 07:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 07:13:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:13:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:13:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:13:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:13:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 07:13:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 07:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 07:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 07:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 07:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 07:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 07:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 07:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 07:13:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 07:13:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 07:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 07:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 07:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 07:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 07:13:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 07:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 07:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 07:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 07:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 07:13:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 76: 0 actions in 0 synapses Jan 4 07:13:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 76 (ref=pe_calc-dc-1325679236-98) derived from /var/lib/pengine/pe-warn-115.bz2 Jan 4 07:13:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 07:13:56 mds1 crmd: [1305]: notice: run_graph: Transition 76 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-115.bz2): Complete Jan 4 07:13:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 76 is now complete Jan 4 07:13:56 mds1 crmd: [1305]: info: notify_crmd: Transition 76 status: done - Jan 4 07:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 07:13:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 07:13:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 76: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-115.bz2 Jan 4 07:13:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 07:17:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:17:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:18:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:18:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:18:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:18:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:19:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:19:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:19:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:19:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:19:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:20:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:20:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:20:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:21:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:21:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:21:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:21:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:21:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:22:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:22:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:22:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:22:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:23:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:23:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:23:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 07:24:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:24:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:24:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:24:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:25:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:25:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:25:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:25:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:25:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:26:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:26:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:26:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:27:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:27:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:27:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:27:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:28:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:28:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:28:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:28:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:28:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:28:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 07:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 07:28:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 07:28:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 07:28:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 118: Requesting the current CIB: S_POLICY_ENGINE Jan 4 07:28:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=118, ref=pe_calc-dc-1325680136-99, seq=4620, quorate=0 Jan 4 07:28:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 07:28:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 07:28:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 07:28:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 07:28:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 07:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 07:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 07:28:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:28:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:28:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:28:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:28:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 07:28:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 07:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 07:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 07:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 07:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 07:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 07:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 07:28:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 07:28:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 07:28:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 07:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 07:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 07:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 07:28:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 07:28:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 07:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 07:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 07:28:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 07:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 07:28:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 77: 0 actions in 0 synapses Jan 4 07:28:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 77 (ref=pe_calc-dc-1325680136-99) derived from /var/lib/pengine/pe-warn-116.bz2 Jan 4 07:28:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 07:28:56 mds1 crmd: [1305]: notice: run_graph: Transition 77 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-116.bz2): Complete Jan 4 07:28:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 77 is now complete Jan 4 07:28:56 mds1 crmd: [1305]: info: notify_crmd: Transition 77 status: done - Jan 4 07:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 07:28:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 07:28:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 77: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-116.bz2 Jan 4 07:28:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 07:29:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:29:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:29:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:29:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:30:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:30:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:30:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:30:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:30:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:31:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:31:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:31:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:31:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:31:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:32:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:32:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:32:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:32:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:32:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:32:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:33:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:33:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:33:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (0.00us average, 0% utilization) in the last 10min Jan 4 07:34:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:34:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:34:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:34:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:35:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:35:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:35:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:35:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:35:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:35:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:36:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:36:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:36:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:36:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:36:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:37:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:37:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:37:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:37:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:38:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:38:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:38:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:38:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:38:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:39:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:39:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:39:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:40:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:40:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:40:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:40:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:41:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:41:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:41:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:41:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:41:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:42:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:42:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:42:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:42:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:43:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:43:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:43:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:43:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:43:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 07:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 07:43:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 07:43:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 07:43:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 119: Requesting the current CIB: S_POLICY_ENGINE Jan 4 07:43:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=119, ref=pe_calc-dc-1325681036-100, seq=4620, quorate=0 Jan 4 07:43:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 07:43:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 07:43:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 07:43:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 07:43:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 07:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 07:43:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 07:43:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:43:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:43:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:43:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:43:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 07:43:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 07:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 07:43:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 07:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 07:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 07:43:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 07:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 07:43:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 07:43:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 07:43:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 07:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 07:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 07:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 07:43:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 07:43:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 07:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 07:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 07:43:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 07:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 07:43:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 78: 0 actions in 0 synapses Jan 4 07:43:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 78 (ref=pe_calc-dc-1325681036-100) derived from /var/lib/pengine/pe-warn-117.bz2 Jan 4 07:43:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 07:43:56 mds1 crmd: [1305]: notice: run_graph: Transition 78 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-117.bz2): Complete Jan 4 07:43:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 78 is now complete Jan 4 07:43:56 mds1 crmd: [1305]: info: notify_crmd: Transition 78 status: done - Jan 4 07:43:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 07:43:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 07:43:56 mds1 pengine: [1304]: WARN: process_pe_message: Transition 78: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-117.bz2 Jan 4 07:43:56 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 07:44:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:44:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:44:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:44:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:44:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:45:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:45:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:45:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:45:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:45:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:46:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:46:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 07:46:28 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 07:46:28 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 07:46:28 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1462 seconds. Jan 4 07:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 07:58:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 07:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 07:58:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 07:58:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 07:58:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 120: Requesting the current CIB: S_POLICY_ENGINE Jan 4 07:58:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=120, ref=pe_calc-dc-1325681936-101, seq=4620, quorate=0 Jan 4 07:58:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 07:58:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 07:58:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 07:58:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 07:58:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 07:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 07:58:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 07:58:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:58:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:58:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:58:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 07:58:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 07:58:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 07:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 07:58:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 07:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 07:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 07:58:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 07:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 07:58:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 07:58:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 07:58:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 07:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 07:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 07:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 07:58:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 07:58:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 07:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 07:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 07:58:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 07:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 07:58:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 79: 0 actions in 0 synapses Jan 4 07:58:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 79 (ref=pe_calc-dc-1325681936-101) derived from /var/lib/pengine/pe-warn-118.bz2 Jan 4 07:58:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 07:58:56 mds1 crmd: [1305]: notice: run_graph: Transition 79 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-118.bz2): Complete Jan 4 07:58:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 79 is now complete Jan 4 07:58:56 mds1 crmd: [1305]: info: notify_crmd: Transition 79 status: done - Jan 4 07:58:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 07:58:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 07:58:57 mds1 pengine: [1304]: WARN: process_pe_message: Transition 79: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-118.bz2 Jan 4 07:58:57 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 08:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 08:10:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:10:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:11:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:11:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:11:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:11:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:12:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:12:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:12:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:12:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:12:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:13:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:13:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:13:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:13:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:13:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:13:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 08:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 08:13:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 08:13:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 08:13:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 121: Requesting the current CIB: S_POLICY_ENGINE Jan 4 08:13:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=121, ref=pe_calc-dc-1325682836-102, seq=4620, quorate=0 Jan 4 08:13:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 08:13:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 08:13:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 08:13:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 08:13:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 08:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 08:13:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 08:13:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:13:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:13:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:13:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:13:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 08:13:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 08:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 08:13:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 08:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 08:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 08:13:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 08:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 08:13:56 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 08:13:56 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 08:13:56 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 08:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 08:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 08:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 08:13:56 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 08:13:56 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 08:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 08:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 08:13:56 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 08:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 08:13:56 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 80: 0 actions in 0 synapses Jan 4 08:13:56 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 80 (ref=pe_calc-dc-1325682836-102) derived from /var/lib/pengine/pe-warn-119.bz2 Jan 4 08:13:56 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 08:13:56 mds1 crmd: [1305]: notice: run_graph: Transition 80 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-119.bz2): Complete Jan 4 08:13:56 mds1 crmd: [1305]: info: te_graph_trigger: Transition 80 is now complete Jan 4 08:13:56 mds1 crmd: [1305]: info: notify_crmd: Transition 80 status: done - Jan 4 08:13:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 08:13:56 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 08:13:57 mds1 pengine: [1304]: WARN: process_pe_message: Transition 80: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-119.bz2 Jan 4 08:13:57 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 08:14:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:14:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:14:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:14:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:14:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:15:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:15:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:15:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:15:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:16:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:16:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:16:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:16:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:17:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:17:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:17:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:18:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:18:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:18:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:18:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:18:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:19:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:19:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:19:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:19:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:20:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:20:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:20:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:20:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:21:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:21:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:21:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:21:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:22:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:22:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:22:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:22:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:22:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:22:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:23:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:23:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:23:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:23:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 08:24:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:24:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:24:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:24:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:24:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:25:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:25:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:25:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:25:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:25:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:26:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:26:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:26:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:26:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:27:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:27:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:27:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:28:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:28:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:28:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:28:56 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 08:28:56 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 08:28:56 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 08:28:56 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 08:28:56 mds1 crmd: [1305]: info: do_pe_invoke: Query 122: Requesting the current CIB: S_POLICY_ENGINE Jan 4 08:28:56 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=122, ref=pe_calc-dc-1325683736-103, seq=4620, quorate=0 Jan 4 08:28:56 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 08:28:56 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 08:28:56 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 08:28:56 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 08:28:56 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 08:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 08:28:56 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 08:28:56 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:28:56 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:28:56 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:28:56 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:28:56 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 08:28:56 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 08:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 08:28:56 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 08:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 08:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 08:28:56 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 08:28:57 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 08:28:57 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 08:28:57 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 08:28:57 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 08:28:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 08:28:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 08:28:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 08:28:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 08:28:57 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 08:28:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 08:28:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 08:28:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 08:28:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 08:28:57 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 81: 0 actions in 0 synapses Jan 4 08:28:57 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 81 (ref=pe_calc-dc-1325683736-103) derived from /var/lib/pengine/pe-warn-120.bz2 Jan 4 08:28:57 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 08:28:57 mds1 crmd: [1305]: notice: run_graph: Transition 81 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-120.bz2): Complete Jan 4 08:28:57 mds1 crmd: [1305]: info: te_graph_trigger: Transition 81 is now complete Jan 4 08:28:57 mds1 crmd: [1305]: info: notify_crmd: Transition 81 status: done - Jan 4 08:28:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 08:28:57 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 08:28:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:28:57 mds1 pengine: [1304]: WARN: process_pe_message: Transition 81: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-120.bz2 Jan 4 08:28:57 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 08:29:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:29:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:29:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:29:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:30:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:30:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:30:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:30:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:31:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:31:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:31:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:31:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:31:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:32:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:32:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:32:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:32:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:32:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:33:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:33:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:33:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:33:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:33:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 08:34:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:34:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:34:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:34:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:35:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:35:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 08:35:37 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 08:35:37 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 08:35:37 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1477 seconds. Jan 4 08:43:57 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 08:43:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 08:43:57 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 08:43:57 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 08:43:57 mds1 crmd: [1305]: info: do_pe_invoke: Query 123: Requesting the current CIB: S_POLICY_ENGINE Jan 4 08:43:57 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=123, ref=pe_calc-dc-1325684637-104, seq=4620, quorate=0 Jan 4 08:43:57 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 08:43:57 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 08:43:57 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 08:43:57 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 08:43:57 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 08:43:57 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 08:43:57 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 08:43:57 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:43:57 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:43:57 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:43:57 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:43:57 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 08:43:57 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 08:43:57 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 08:43:57 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 08:43:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 08:43:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 08:43:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 08:43:57 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 08:43:57 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 08:43:57 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 08:43:57 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 08:43:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 08:43:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 08:43:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 08:43:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 08:43:57 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 08:43:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 08:43:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 08:43:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 08:43:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 08:43:57 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 82: 0 actions in 0 synapses Jan 4 08:43:57 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 82 (ref=pe_calc-dc-1325684637-104) derived from /var/lib/pengine/pe-warn-121.bz2 Jan 4 08:43:57 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 08:43:57 mds1 crmd: [1305]: notice: run_graph: Transition 82 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-121.bz2): Complete Jan 4 08:43:57 mds1 crmd: [1305]: info: te_graph_trigger: Transition 82 is now complete Jan 4 08:43:57 mds1 crmd: [1305]: info: notify_crmd: Transition 82 status: done - Jan 4 08:43:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 08:43:57 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 08:43:57 mds1 pengine: [1304]: WARN: process_pe_message: Transition 82: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-121.bz2 Jan 4 08:43:57 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 08:53:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 08:58:57 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 08:58:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 08:58:57 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 08:58:57 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 08:58:57 mds1 crmd: [1305]: info: do_pe_invoke: Query 124: Requesting the current CIB: S_POLICY_ENGINE Jan 4 08:58:57 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=124, ref=pe_calc-dc-1325685537-105, seq=4620, quorate=0 Jan 4 08:58:57 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 08:58:57 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 08:58:57 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 08:58:57 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 08:58:57 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 08:58:57 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 08:58:57 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 08:58:57 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:58:57 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:58:57 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:58:57 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 08:58:57 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 08:58:57 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 08:58:57 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 08:58:57 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 08:58:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 08:58:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 08:58:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 08:58:57 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 08:58:57 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 08:58:57 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 08:58:57 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 08:58:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 08:58:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 08:58:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 08:58:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 08:58:57 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 08:58:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 08:58:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 08:58:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 08:58:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 08:58:57 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 83: 0 actions in 0 synapses Jan 4 08:58:57 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 83 (ref=pe_calc-dc-1325685537-105) derived from /var/lib/pengine/pe-warn-122.bz2 Jan 4 08:58:57 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 08:58:57 mds1 crmd: [1305]: notice: run_graph: Transition 83 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-122.bz2): Complete Jan 4 08:58:57 mds1 crmd: [1305]: info: te_graph_trigger: Transition 83 is now complete Jan 4 08:58:57 mds1 crmd: [1305]: info: notify_crmd: Transition 83 status: done - Jan 4 08:58:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 08:58:57 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 08:58:57 mds1 pengine: [1304]: WARN: process_pe_message: Transition 83: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-122.bz2 Jan 4 08:58:57 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 09:00:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:00:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:00:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:00:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:00:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:01:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:01:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:01:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:01:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:02:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:02:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:02:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:02:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:02:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:03:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:03:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:03:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:03:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:03:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 09:03:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:04:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:04:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:04:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:04:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:05:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:05:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:05:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:05:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:05:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:06:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:06:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:06:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:06:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:07:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:07:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:07:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:07:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:08:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:08:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:08:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:08:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:09:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:09:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:09:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:09:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:10:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:10:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:10:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:10:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:11:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:11:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:11:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:11:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:12:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:12:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:12:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:12:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:13:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:13:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:13:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:13:57 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 09:13:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 09:13:57 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 09:13:57 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 09:13:57 mds1 crmd: [1305]: info: do_pe_invoke: Query 125: Requesting the current CIB: S_POLICY_ENGINE Jan 4 09:13:57 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=125, ref=pe_calc-dc-1325686437-106, seq=4620, quorate=0 Jan 4 09:13:57 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 09:13:57 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 09:13:57 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 09:13:57 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 09:13:57 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 09:13:57 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 09:13:57 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 09:13:57 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 09:13:57 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 09:13:57 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 09:13:57 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 09:13:57 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 09:13:57 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 09:13:57 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 09:13:57 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 09:13:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 09:13:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 09:13:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 09:13:57 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 09:13:57 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 09:13:57 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 09:13:57 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 09:13:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 09:13:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 09:13:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 09:13:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 09:13:57 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 09:13:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 09:13:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 09:13:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 09:13:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 09:13:57 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 84: 0 actions in 0 synapses Jan 4 09:13:57 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 84 (ref=pe_calc-dc-1325686437-106) derived from /var/lib/pengine/pe-warn-123.bz2 Jan 4 09:13:57 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 09:13:57 mds1 crmd: [1305]: notice: run_graph: Transition 84 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-123.bz2): Complete Jan 4 09:13:57 mds1 crmd: [1305]: info: te_graph_trigger: Transition 84 is now complete Jan 4 09:13:57 mds1 crmd: [1305]: info: notify_crmd: Transition 84 status: done - Jan 4 09:13:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 09:13:57 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 09:13:57 mds1 pengine: [1304]: WARN: process_pe_message: Transition 84: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-123.bz2 Jan 4 09:13:57 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 09:14:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:14:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:14:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:14:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:14:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:15:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:15:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:15:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:15:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:16:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:16:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:16:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:16:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:16:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:17:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:17:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:17:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:17:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:17:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:18:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:18:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:18:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:18:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:19:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:19:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:19:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:19:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:19:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:20:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:20:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:20:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:21:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:21:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:21:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:21:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:21:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:22:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:22:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:22:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:22:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:23:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:23:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:23:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:23:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:23:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:23:55 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 09:23:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:24:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:24:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:24:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:24:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:25:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 09:25:24 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 09:25:24 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 09:25:24 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1267 seconds. Jan 4 09:28:57 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 09:28:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 09:28:57 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 09:28:57 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 09:28:57 mds1 crmd: [1305]: info: do_pe_invoke: Query 126: Requesting the current CIB: S_POLICY_ENGINE Jan 4 09:28:57 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=126, ref=pe_calc-dc-1325687337-107, seq=4620, quorate=0 Jan 4 09:28:57 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 09:28:57 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 09:28:57 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 09:28:57 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 09:28:57 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 09:28:57 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 09:28:57 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 09:28:57 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 09:28:57 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 09:28:57 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 09:28:57 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 09:28:57 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 09:28:57 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 09:28:57 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 09:28:57 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 09:28:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 09:28:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 09:28:57 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 09:28:57 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 09:28:57 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 09:28:57 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 09:28:57 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 09:28:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 09:28:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 09:28:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 09:28:57 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 09:28:57 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 09:28:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 09:28:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 09:28:57 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 09:28:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 09:28:57 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 85: 0 actions in 0 synapses Jan 4 09:28:57 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 85 (ref=pe_calc-dc-1325687337-107) derived from /var/lib/pengine/pe-warn-124.bz2 Jan 4 09:28:57 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 09:28:57 mds1 crmd: [1305]: notice: run_graph: Transition 85 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-124.bz2): Complete Jan 4 09:28:57 mds1 crmd: [1305]: info: te_graph_trigger: Transition 85 is now complete Jan 4 09:28:57 mds1 crmd: [1305]: info: notify_crmd: Transition 85 status: done - Jan 4 09:28:57 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 09:28:57 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 09:28:57 mds1 pengine: [1304]: WARN: process_pe_message: Transition 85: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-124.bz2 Jan 4 09:28:57 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 10:01:01 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 10:01:01 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 10:01:01 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 10:01:01 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 10:01:01 mds1 crmd: [1305]: info: do_state_transition: All 1 cluster nodes are eligible to run resources. Jan 4 10:01:01 mds1 crmd: [1305]: info: do_pe_invoke: Query 127: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:01:01 mds1 corosync[1294]: [TOTEM ] A processor failed, forming new configuration. Jan 4 10:01:01 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4628: memb=1, new=0, lost=0 Jan 4 10:01:01 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 10:01:01 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4628: memb=1, new=0, lost=0 Jan 4 10:01:01 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 10:01:01 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 10:01:01 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=127, ref=pe_calc-dc-1325689261-108, seq=4620, quorate=0 Jan 4 10:01:01 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:01:01 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:01:01 mds1 corosync[1294]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 10:01:01 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:01:01 mds1 corosync[1294]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 10:01:01 mds1 pengine: [1304]: WARN: cluster_status: We do not have quorum - fencing and resource management disabled Jan 4 10:01:01 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:01:01 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:01:01 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:01:01 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:01:01 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:01:01 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:01:01 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:01:01 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:01:01 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 10:01:01 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 10:01:01 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 10:01:01 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 10:01:01 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:01:01 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:01:01 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:01:01 mds1 pengine: [1304]: WARN: stage6: Node oss1 is unclean! Jan 4 10:01:01 mds1 pengine: [1304]: WARN: stage6: Node oss2 is unclean! Jan 4 10:01:01 mds1 pengine: [1304]: WARN: stage6: Node mds2 is unclean! Jan 4 10:01:01 mds1 pengine: [1304]: notice: stage6: Cannot fence unclean nodes until quorum is attained (or no-quorum-policy is set to ignore) Jan 4 10:01:01 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:01:01 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:01:01 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds1) Jan 4 10:01:01 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:01:01 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 10:01:01 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 10:01:01 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(mds1) Jan 4 10:01:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:01:01 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(mds1) Jan 4 10:01:01 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 10:01:01 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 86: 0 actions in 0 synapses Jan 4 10:01:01 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 86 (ref=pe_calc-dc-1325689261-108) derived from /var/lib/pengine/pe-warn-125.bz2 Jan 4 10:01:01 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 10:01:01 mds1 crmd: [1305]: notice: run_graph: Transition 86 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-warn-125.bz2): Complete Jan 4 10:01:01 mds1 crmd: [1305]: info: te_graph_trigger: Transition 86 is now complete Jan 4 10:01:01 mds1 crmd: [1305]: info: notify_crmd: Transition 86 status: done - Jan 4 10:01:01 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 10:01:01 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 10:01:01 mds1 pengine: [1304]: WARN: process_pe_message: Transition 86: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-125.bz2 Jan 4 10:01:01 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 10:01:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:01:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:01:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:01:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:01:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:02:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:02:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:02:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:02:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:03:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:03:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:03:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:03:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:04:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:04:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:04:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:04:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:05:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:05:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:05:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:05:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:06:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:06:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:06:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:07:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:07:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:07:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:07:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:08:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:08:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:08:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:08:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:09:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:09:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:09:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:10:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:10:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:10:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:10:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:10:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:11:01 mds1 cib: [1301]: info: cib_stats: Processed 1 operations (10000.00us average, 0% utilization) in the last 10min Jan 4 10:11:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:11:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:11:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:11:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:12:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:12:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:12:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:12:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:13:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4672: memb=1, new=0, lost=0 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4672: memb=2, new=1, lost=0 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: update_member: Creating entry for node 2474289344 born on 4672 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: update_member: Node 2474289344/unknown is now: member Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2474289344 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2474289344 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 4672 to 2 children Jan 4 10:13:13 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 10:13:13 mds1 cib: [1301]: info: ais_dispatch_message: Membership 4672: quorum still lost Jan 4 10:13:13 mds1 cib: [1301]: info: crm_new_peer: Node now has id: 2474289344 Jan 4 10:13:13 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 4672: quorum still lost Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: update_member: 0xee5ad0 Node 2474289344 (oss1) born on: 4672 Jan 4 10:13:13 mds1 cib: [1301]: info: crm_update_peer: Node (null): id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=0 born=0 seen=4672 proc=00000000000000000000000000000000 Jan 4 10:13:13 mds1 crmd: [1305]: info: crm_new_peer: Node now has id: 2474289344 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: update_member: 0xee5ad0 Node 2474289344 now known as oss1 (was: (null)) Jan 4 10:13:13 mds1 cib: [1301]: info: ais_dispatch_message: Membership 4672: quorum still lost Jan 4 10:13:13 mds1 crmd: [1305]: info: crm_update_peer: Node (null): id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=0 born=0 seen=4672 proc=00000000000000000000000000000000 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: update_member: Node oss1 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 10:13:13 mds1 cib: [1301]: info: crm_get_peer: Node 2474289344 is now known as oss1 Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: update_member: Node oss1 now has 1 quorum votes (was 0) Jan 4 10:13:13 mds1 cib: [1301]: info: crm_update_peer: Node oss1: id=2474289344 state=member addr=r(0) ip(192.168.122.147) votes=1 (new) born=4672 seen=4672 proc=00000000000000000000000000111312 (new) Jan 4 10:13:13 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 4672 to 2 children Jan 4 10:13:13 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/128, version=0.232.16): ok (rc=0) Jan 4 10:13:13 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:13 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/131, version=0.232.16): ok (rc=0) Jan 4 10:13:13 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 4672: quorum still lost Jan 4 10:13:13 mds1 crmd: [1305]: info: crm_get_peer: Node 2474289344 is now known as oss1 Jan 4 10:13:13 mds1 crmd: [1305]: info: ais_status_callback: status: oss1 is now member Jan 4 10:13:13 mds1 crmd: [1305]: notice: crmd_peer_update: Status update: Client oss1/crmd now has status [online] (DC=true) Jan 4 10:13:13 mds1 crmd: [1305]: info: crm_update_peer: Node oss1: id=2474289344 state=member addr=r(0) ip(192.168.122.147) votes=1 (new) born=4672 seen=4672 proc=00000000000000000000000000111312 (new) Jan 4 10:13:13 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/133, version=0.232.17): ok (rc=0) Jan 4 10:13:13 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:13 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/136, version=0.232.18): ok (rc=0) Jan 4 10:13:13 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:13 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=crmd_peer_update ] Jan 4 10:13:13 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:13 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:184 - Triggered transition abort (complete=1) : Peer Halt Jan 4 10:13:13 mds1 corosync[1294]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 10:13:13 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:13 mds1 corosync[1294]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 10:13:13 mds1 crmd: [1305]: info: join_make_offer: Making join offers based on membership 4672 Jan 4 10:13:13 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-2: Waiting on 2 outstanding join acks Jan 4 10:13:13 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:15 mds1 corosync[1294]: [TOTEM ] A processor failed, forming new configuration. Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4676: memb=1, new=0, lost=1 Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: lost: oss1 2474289344 Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4676: memb=1, new=0, lost=0 Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] info: ais_mark_unseen_peer_dead: Node oss1 was not seen in the previous transition Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] info: update_member: Node 2474289344/oss1 is now: lost Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 4676 to 2 children Jan 4 10:13:16 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 10:13:16 mds1 corosync[1294]: [CPG ] downlist received left_list: 1 Jan 4 10:13:16 mds1 corosync[1294]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 10:13:16 mds1 corosync[1294]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 10:13:16 mds1 cib: [1301]: info: ais_dispatch_message: Membership 4676: quorum still lost Jan 4 10:13:16 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 4676: quorum still lost Jan 4 10:13:16 mds1 cib: [1301]: info: crm_update_peer: Node oss1: id=2474289344 state=lost (new) addr=r(0) ip(192.168.122.147) votes=1 born=4672 seen=4672 proc=00000000000000000000000000111312 Jan 4 10:13:16 mds1 crmd: [1305]: info: ais_status_callback: status: oss1 is now lost (was member) Jan 4 10:13:16 mds1 crmd: [1305]: info: crm_update_peer: Node oss1: id=2474289344 state=lost (new) addr=r(0) ip(192.168.122.147) votes=1 born=4672 seen=4672 proc=00000000000000000000000000111312 Jan 4 10:13:16 mds1 crmd: [1305]: info: erase_node_from_join: Removed node oss1 from join calculations: welcomed=1 itegrated=0 finalized=0 confirmed=0 Jan 4 10:13:16 mds1 crmd: [1305]: info: check_join_state: check_dead_member: Membership changed since join started: 4672 -> 4676 Jan 4 10:13:16 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/138, version=0.232.18): ok (rc=0) Jan 4 10:13:16 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:16 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/141, version=0.232.19): ok (rc=0) Jan 4 10:13:16 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:16 mds1 crmd: [1305]: info: join_make_offer: Making join offers based on membership 4676 Jan 4 10:13:16 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-3: Waiting on 1 outstanding join acks Jan 4 10:13:16 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4680: memb=1, new=0, lost=0 Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4680: memb=2, new=1, lost=0 Jan 4 10:13:16 mds1 corosync[1294]: [pcmk ] info: update_member: Node 2474289344/oss1 is now: member Jan 4 10:13:17 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: NEW: oss1 2474289344 Jan 4 10:13:17 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 10:13:17 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: oss1 2474289344 Jan 4 10:13:17 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 4680 to 2 children Jan 4 10:13:17 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 10:13:17 mds1 cib: [1301]: info: ais_dispatch_message: Membership 4680: quorum still lost Jan 4 10:13:17 mds1 cib: [1301]: info: crm_update_peer: Node oss1: id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=1 born=4672 seen=4680 proc=00000000000000000000000000111312 Jan 4 10:13:17 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 4680: quorum still lost Jan 4 10:13:17 mds1 crmd: [1305]: info: ais_status_callback: status: oss1 is now member (was lost) Jan 4 10:13:17 mds1 crmd: [1305]: info: crm_update_peer: Node oss1: id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=1 born=4672 seen=4680 proc=00000000000000000000000000111312 Jan 4 10:13:17 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/lrm (origin=local/crmd/143, version=0.232.19): ok (rc=0) Jan 4 10:13:17 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/transient_attributes (origin=local/crmd/144, version=0.232.19): ok (rc=0) Jan 4 10:13:17 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/145, version=0.232.19): ok (rc=0) Jan 4 10:13:17 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:17 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/148, version=0.232.20): ok (rc=0) Jan 4 10:13:17 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss1']/lrm": ok (rc=0) Jan 4 10:13:17 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss1']/transient_attributes": ok (rc=0) Jan 4 10:13:17 mds1 crmd: [1305]: info: check_join_state: do_dc_join_filter_offer: Membership changed since join started: 4676 -> 4680 Jan 4 10:13:17 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:17 mds1 crmd: [1305]: info: join_make_offer: Making join offers based on membership 4680 Jan 4 10:13:17 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-4: Waiting on 2 outstanding join acks Jan 4 10:13:17 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:17 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:17 mds1 corosync[1294]: [CPG ] downlist received left_list: 1 Jan 4 10:13:17 mds1 corosync[1294]: [CPG ] chosen downlist from node r(0) ip(192.168.122.147) Jan 4 10:13:17 mds1 corosync[1294]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 10:13:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:13:20 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:20 mds1 crmd: [1305]: info: do_dc_join_offer_all: A new node joined the cluster Jan 4 10:13:20 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-5: Waiting on 2 outstanding join acks Jan 4 10:13:20 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 4984: memb=2, new=0, lost=0 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: oss1 2474289344 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 4984: memb=3, new=1, lost=0 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: update_member: Creating entry for node 2608507072 born on 4984 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: update_member: Node 2608507072/unknown is now: member Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2608507072 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: oss1 2474289344 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2608507072 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 4984 to 2 children Jan 4 10:13:21 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 10:13:21 mds1 cib: [1301]: info: ais_dispatch_message: Membership 4984: quorum still lost Jan 4 10:13:21 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 4984: quorum still lost Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: update_member: 0xed3a60 Node 2608507072 (mds2) born on: 4984 Jan 4 10:13:21 mds1 cib: [1301]: info: crm_new_peer: Node now has id: 2608507072 Jan 4 10:13:21 mds1 crmd: [1305]: info: crm_new_peer: Node now has id: 2608507072 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: update_member: 0xed3a60 Node 2608507072 now known as mds2 (was: (null)) Jan 4 10:13:21 mds1 cib: [1301]: info: crm_update_peer: Node (null): id=2608507072 state=member (new) addr=r(0) ip(192.168.122.155) votes=0 born=0 seen=4984 proc=00000000000000000000000000000000 Jan 4 10:13:21 mds1 crmd: [1305]: info: crm_update_peer: Node (null): id=2608507072 state=member (new) addr=r(0) ip(192.168.122.155) votes=0 born=0 seen=4984 proc=00000000000000000000000000000000 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: update_member: Node mds2 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 10:13:21 mds1 cib: [1301]: notice: ais_dispatch_message: Membership 4984: quorum acquired Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: update_member: Node mds2 now has 1 quorum votes (was 0) Jan 4 10:13:21 mds1 cib: [1301]: info: crm_get_peer: Node 2608507072 is now known as mds2 Jan 4 10:13:21 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 4984 to 2 children Jan 4 10:13:21 mds1 cib: [1301]: info: crm_update_peer: Node mds2: id=2608507072 state=member addr=r(0) ip(192.168.122.155) votes=1 (new) born=4984 seen=4984 proc=00000000000000000000000000111312 (new) Jan 4 10:13:21 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/151, version=0.232.20): ok (rc=0) Jan 4 10:13:21 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:21 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/154, version=0.232.20): ok (rc=0) Jan 4 10:13:21 mds1 crmd: [1305]: notice: ais_dispatch_message: Membership 4984: quorum acquired Jan 4 10:13:21 mds1 crmd: [1305]: info: crm_get_peer: Node 2608507072 is now known as mds2 Jan 4 10:13:21 mds1 crmd: [1305]: info: ais_status_callback: status: mds2 is now member Jan 4 10:13:21 mds1 crmd: [1305]: notice: crmd_peer_update: Status update: Client mds2/crmd now has status [online] (DC=true) Jan 4 10:13:21 mds1 crmd: [1305]: info: crm_update_peer: Node mds2: id=2608507072 state=member addr=r(0) ip(192.168.122.155) votes=1 (new) born=4984 seen=4984 proc=00000000000000000000000000111312 (new) Jan 4 10:13:21 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/156, version=0.232.21): ok (rc=0) Jan 4 10:13:21 mds1 crmd: [1305]: info: crm_update_quorum: Updating quorum status to true (call=158) Jan 4 10:13:21 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:21 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:21 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:21 mds1 corosync[1294]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 10:13:21 mds1 corosync[1294]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 10:13:22 mds1 cib: [1301]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-74.raw Jan 4 10:13:22 mds1 cib: [1301]: info: write_cib_contents: Wrote version 0.233.0 of the CIB to disk (digest: 337a556220aa97020158f0911898b9d0) Jan 4 10:13:22 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.paPgTg (digest: /var/lib/heartbeat/crm/cib.0xC3iF) Jan 4 10:13:22 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:22 mds1 cib: [1301]: info: log_data_element: cib:diff: + Jan 4 10:13:22 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/158, version=0.233.1): ok (rc=0) Jan 4 10:13:22 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:22 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:22 mds1 crmd: [1305]: info: join_make_offer: Making join offers based on membership 4984 Jan 4 10:13:22 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/160, version=0.233.1): ok (rc=0) Jan 4 10:13:22 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-6: Waiting on 3 outstanding join acks Jan 4 10:13:22 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:24 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:24 mds1 crmd: [1305]: info: do_dc_join_offer_all: A new node joined the cluster Jan 4 10:13:24 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-7: Waiting on 3 outstanding join acks Jan 4 10:13:24 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:25 mds1 crmd: [1305]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 10:13:25 mds1 crmd: [1305]: info: do_state_transition: All 3 cluster nodes responded to the join offer. Jan 4 10:13:25 mds1 crmd: [1305]: info: do_dc_join_finalize: join-7: Syncing the CIB from mds2 to the rest of the cluster Jan 4 10:13:25 mds1 cib: [1301]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-75.raw Jan 4 10:13:25 mds1 cib: [1301]: info: write_cib_contents: Wrote version 0.243.0 of the CIB to disk (digest: 6091f4e89fa3415ec87955bfa350fd80) Jan 4 10:13:25 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.lmoOz9 (digest: /var/lib/heartbeat/crm/cib.q6jzQD) Jan 4 10:13:25 mds1 cib: [1301]: info: cib_replace_notify: Local-only Replace: -1.-1.-1 from mds2 Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 attrd: [1303]: info: do_cib_replaced: Sending full refresh Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:25 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: + Jan 4 10:13:26 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/164, version=0.243.0): ok (rc=0) Jan 4 10:13:26 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Jan 4 10:13:26 mds1 lrmd: [1302]: info: stonith_api_device_metadata: looking up fence_virsh/redhat metadata Jan 4 10:13:26 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/165, version=0.243.0): ok (rc=0) Jan 4 10:13:26 mds1 lrmd: [1302]: info: stonithRA plugin: got metadata: #012#012fence_virsh is an I/O Fencing agent which can be used with the virtual machines managed by libvirt. It logs via ssh to a dom0 and there run virsh command, which does all work. #012.P#012By default, virsh needs root account to do properly work. So you must allow ssh login in your sshd_config.#012#012#011#012#011#011#012#011#011#012#011#011Fencing Action#012#011#012#011#012#011#011#012#011#011#012#011#011IP Address or Hostname#012#011#012#011#012#011#011#012#011#011#012#011#011Login Name#012#011#012#011#012#011#011#012#011#011#012#011#011Login password or passphrase#012#011#012#011#012#011#011#012#011#011#012#011#011Script to retrieve password#012#011#012#011#012#011#011#012#011#011#012#011#011SSH connection#012#011#012#011#012#011#011 S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 10:13:26 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/transient_attributes (origin=oss1/crmd/8, version=0.243.4): ok (rc=0) Jan 4 10:13:26 mds1 attrd: [1303]: info: crm_new_peer: Node 2608507072 is now known as mds2 Jan 4 10:13:26 mds1 crmd: [1305]: info: do_state_transition: All 3 cluster nodes are eligible to run resources. Jan 4 10:13:26 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/173, version=0.243.4): ok (rc=0) Jan 4 10:13:26 mds1 crmd: [1305]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date Jan 4 10:13:26 mds1 attrd: [1303]: info: crm_new_peer: Node oss1 now has id: 2474289344 Jan 4 10:13:26 mds1 crmd: [1305]: info: crm_update_quorum: Updating quorum status to true (call=175) Jan 4 10:13:26 mds1 attrd: [1303]: info: crm_new_peer: Node 2474289344 is now known as oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:176 - Triggered transition abort (complete=1) : Peer Cancelled Jan 4 10:13:26 mds1 crmd: [1305]: info: do_pe_invoke: Query 176: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:13:26 mds1 cib: [1301]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-76.raw Jan 4 10:13:26 mds1 cib: [1301]: info: write_cib_contents: Wrote version 0.244.0 of the CIB to disk (digest: 02c0e004c66711b1a64b9a53ae16b3e1) Jan 4 10:13:26 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.0gGLH9 (digest: /var/lib/heartbeat/crm/cib.EkaYyF) Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:26 mds1 cib: [1301]: info: log_data_element: cib:diff: + Jan 4 10:13:26 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/175, version=0.244.1): ok (rc=0) Jan 4 10:13:26 mds1 crmd: [1305]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change Jan 4 10:13:26 mds1 crmd: [1305]: info: need_abort: Aborting on change to have-quorum Jan 4 10:13:26 mds1 crmd: [1305]: info: do_pe_invoke: Query 177: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:13:26 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=177, ref=pe_calc-dc-1325690006-138, seq=4984, quorate=1 Jan 4 10:13:26 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:13:26 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:13:26 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:13:26 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:13:26 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:13:26 mds1 attrd: [1303]: info: attrd_local_callback: Sending full refresh (origin=crmd) Jan 4 10:13:26 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:13:26 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Jan 4 10:13:26 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:13:26 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Jan 4 10:13:26 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:13:26 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Jan 4 10:13:26 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:26 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:26 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:26 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:26 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 10:13:26 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 10:13:26 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 10:13:26 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 10:13:26 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:13:26 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:13:26 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:13:26 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:13:26 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 10:13:26 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 10:13:26 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:13:26 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:13:26 mds1 pengine: [1304]: WARN: stage6: Scheduling Node oss2 for STONITH Jan 4 10:13:26 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:13:26 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:13:26 mds1 pengine: [1304]: notice: LogActions: Move resource st-mds1#011(Started mds1 -> oss1) Jan 4 10:13:26 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:13:26 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 10:13:26 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 10:13:26 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(oss1) Jan 4 10:13:26 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(oss1) Jan 4 10:13:26 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 10:13:26 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 87: 33 actions in 33 synapses Jan 4 10:13:26 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 87 (ref=pe_calc-dc-1325690006-138) derived from /var/lib/pengine/pe-warn-126.bz2 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 4: monitor st-oss2_monitor_0 on oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 13: monitor st-oss2_monitor_0 on mds2 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 5: monitor st-oss1_monitor_0 on oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 14: monitor st-oss1_monitor_0 on mds2 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 6: monitor st-mds1_monitor_0 on oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 15: monitor st-mds1_monitor_0 on mds2 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 7: monitor st-mds2_monitor_0 on oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 16: monitor st-mds2_monitor_0 on mds2 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 8: monitor MGS_1_monitor_0 on oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 17: monitor MGS_1_monitor_0 on mds2 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 9: monitor lustre-MDT0000_2_monitor_0 on oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 18: monitor lustre-MDT0000_2_monitor_0 on mds2 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 10: monitor lustre-OST0001_4_monitor_0 on oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 19: monitor lustre-OST0001_4_monitor_0 on mds2 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 11: monitor lustre-OST0000_3_monitor_0 on oss1 Jan 4 10:13:26 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 20: monitor lustre-OST0000_3_monitor_0 on mds2 Jan 4 10:13:26 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5020: memb=3, new=0, lost=0 Jan 4 10:13:26 mds1 cib: [1301]: info: ais_dispatch_message: Membership 5020: quorum retained Jan 4 10:13:26 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5020: quorum retained Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 10:13:27 mds1 cib: [1301]: info: crm_new_peer: Node now has id: 2692393152 Jan 4 10:13:27 mds1 crmd: [1305]: info: crm_new_peer: Node now has id: 2692393152 Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: oss1 2474289344 Jan 4 10:13:27 mds1 cib: [1301]: info: crm_update_peer: Node (null): id=2692393152 state=member (new) addr=r(0) ip(192.168.122.160) votes=0 born=0 seen=5020 proc=00000000000000000000000000000000 Jan 4 10:13:27 mds1 crmd: [1305]: info: crm_update_peer: Node (null): id=2692393152 state=member (new) addr=r(0) ip(192.168.122.160) votes=0 born=0 seen=5020 proc=00000000000000000000000000000000 Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds2 2608507072 Jan 4 10:13:27 mds1 cib: [1301]: info: ais_dispatch_message: Membership 5020: quorum retained Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5020: memb=4, new=1, lost=0 Jan 4 10:13:27 mds1 cib: [1301]: info: crm_get_peer: Node 2692393152 is now known as oss2 Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: update_member: Creating entry for node 2692393152 born on 5020 Jan 4 10:13:27 mds1 cib: [1301]: info: crm_update_peer: Node oss2: id=2692393152 state=member addr=r(0) ip(192.168.122.160) votes=1 (new) born=5020 seen=5020 proc=00000000000000000000000000111312 (new) Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: update_member: Node 2692393152/unknown is now: member Jan 4 10:13:27 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/178, version=0.244.1): ok (rc=0) Jan 4 10:13:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2692393152 Jan 4 10:13:27 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:27 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/181, version=0.244.1): ok (rc=0) Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 10:13:27 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5020: quorum retained Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: oss1 2474289344 Jan 4 10:13:27 mds1 crmd: [1305]: info: crm_get_peer: Node 2692393152 is now known as oss2 Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds2 2608507072 Jan 4 10:13:27 mds1 crmd: [1305]: info: ais_status_callback: status: oss2 is now member Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2692393152 Jan 4 10:13:27 mds1 crmd: [1305]: notice: crmd_peer_update: Status update: Client oss2/crmd now has status [online] (DC=true) Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 5020 to 2 children Jan 4 10:13:27 mds1 crmd: [1305]: info: crm_update_peer: Node oss2: id=2692393152 state=member addr=r(0) ip(192.168.122.160) votes=1 (new) born=5020 seen=5020 proc=00000000000000000000000000111312 (new) Jan 4 10:13:27 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 10:13:27 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/183, version=0.244.2): ok (rc=0) Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: update_member: 0xee5a00 Node 2692393152 (oss2) born on: 5020 Jan 4 10:13:27 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:27 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/186, version=0.244.3): ok (rc=0) Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: update_member: 0xee5a00 Node 2692393152 now known as oss2 (was: (null)) Jan 4 10:13:27 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=crmd_peer_update ] Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: update_member: Node oss2 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 10:13:27 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:184 - Triggered transition abort (complete=0) : Peer Halt Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: update_member: Node oss2 now has 1 quorum votes (was 0) Jan 4 10:13:27 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Jan 4 10:13:27 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 5020 to 2 children Jan 4 10:13:27 mds1 crmd: [1305]: info: update_abort_priority: Abort action done superceeded by stop Jan 4 10:13:27 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:184 - Triggered transition abort (complete=0) : Peer Halt Jan 4 10:13:27 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:184 - Triggered transition abort (complete=0) : Peer Halt Jan 4 10:13:27 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:184 - Triggered transition abort (complete=0) : Peer Halt Jan 4 10:13:27 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:184 - Triggered transition abort (complete=0) : Peer Halt Jan 4 10:13:27 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:27 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:27 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:27 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:27 mds1 corosync[1294]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 10:13:27 mds1 corosync[1294]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 10:13:27 mds1 pengine: [1304]: WARN: process_pe_message: Transition 87: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-warn-126.bz2 Jan 4 10:13:27 mds1 pengine: [1304]: info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. Jan 4 10:13:27 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_monitor_0 (7) confirmed on oss1 (rc=0) Jan 4 10:13:27 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_monitor_0 (4) confirmed on oss1 (rc=0) Jan 4 10:13:27 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_monitor_0 (5) confirmed on oss1 (rc=0) Jan 4 10:13:27 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_monitor_0 (6) confirmed on oss1 (rc=0) Jan 4 10:13:28 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (11) confirmed on oss1 (rc=0) Jan 4 10:13:28 mds1 crmd: [1305]: WARN: status_from_rc: Action 9 (lustre-MDT0000_2_monitor_0) on oss1 failed (target: 7 vs. rc: 0): Error Jan 4 10:13:28 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_monitor_0, magic=0:0;9:87:7:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.244.9) : Event failed Jan 4 10:13:28 mds1 crmd: [1305]: info: update_abort_priority: Abort action stop superceeded by restart Jan 4 10:13:28 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (9) confirmed on oss1 (rc=4) Jan 4 10:13:28 mds1 crmd: [1305]: WARN: status_from_rc: Action 8 (MGS_1_monitor_0) on oss1 failed (target: 7 vs. rc: 0): Error Jan 4 10:13:28 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_monitor_0, magic=0:0;8:87:7:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.244.10) : Event failed Jan 4 10:13:28 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_monitor_0 (8) confirmed on oss1 (rc=4) Jan 4 10:13:28 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (10) confirmed on oss1 (rc=0) Jan 4 10:13:28 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 3: probe_complete probe_complete on oss1 - no waiting Jan 4 10:13:28 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_monitor_0 (15) confirmed on mds2 (rc=0) Jan 4 10:13:28 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_monitor_0 (14) confirmed on mds2 (rc=0) Jan 4 10:13:28 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_monitor_0 (16) confirmed on mds2 (rc=0) Jan 4 10:13:28 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_monitor_0 (13) confirmed on mds2 (rc=0) Jan 4 10:13:28 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.244.16) : Transient attribute: update Jan 4 10:13:29 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_monitor_0 (17) confirmed on mds2 (rc=0) Jan 4 10:13:29 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (18) confirmed on mds2 (rc=0) Jan 4 10:13:29 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_sync_one for section 'all' (origin=oss2/oss2/(null), version=0.244.18): ok (rc=0) Jan 4 10:13:29 mds1 crmd: [1305]: WARN: status_from_rc: Action 20 (lustre-OST0000_3_monitor_0) on mds2 failed (target: 7 vs. rc: 0): Error Jan 4 10:13:29 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0000_3_monitor_0, magic=0:0;20:87:7:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.244.19) : Event failed Jan 4 10:13:29 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (20) confirmed on mds2 (rc=4) Jan 4 10:13:29 mds1 crmd: [1305]: WARN: status_from_rc: Action 19 (lustre-OST0001_4_monitor_0) on mds2 failed (target: 7 vs. rc: 0): Error Jan 4 10:13:29 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0001_4_monitor_0, magic=0:0;19:87:7:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.244.20) : Event failed Jan 4 10:13:29 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (19) confirmed on mds2 (rc=4) Jan 4 10:13:29 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 12: probe_complete probe_complete on mds2 - no waiting Jan 4 10:13:29 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 10:13:29 mds1 crmd: [1305]: notice: run_graph: Transition 87 (Complete=18, Pending=0, Fired=0, Skipped=15, Incomplete=0, Source=/var/lib/pengine/pe-warn-126.bz2): Stopped Jan 4 10:13:29 mds1 crmd: [1305]: info: te_graph_trigger: Transition 87 is now complete Jan 4 10:13:29 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:184 - Triggered transition abort (complete=1) : Peer Halt Jan 4 10:13:29 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:29 mds1 crmd: [1305]: info: join_make_offer: Making join offers based on membership 5020 Jan 4 10:13:29 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-8: Waiting on 4 outstanding join acks Jan 4 10:13:29 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:31 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:31 mds1 crmd: [1305]: info: do_dc_join_offer_all: A new node joined the cluster Jan 4 10:13:31 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-9: Waiting on 4 outstanding join acks Jan 4 10:13:31 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:32 mds1 attrd: [1303]: info: crm_new_peer: Node oss2 now has id: 2692393152 Jan 4 10:13:32 mds1 attrd: [1303]: info: crm_new_peer: Node 2692393152 is now known as oss2 Jan 4 10:13:32 mds1 crmd: [1305]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 10:13:32 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes responded to the join offer. Jan 4 10:13:32 mds1 crmd: [1305]: info: do_dc_join_finalize: join-9: Syncing the CIB from mds1 to the rest of the cluster Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/189, version=0.244.21): ok (rc=0) Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/190, version=0.244.21): ok (rc=0) Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/191, version=0.244.21): ok (rc=0) Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/192, version=0.244.21): ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: do_dc_join_ack: join-9: Updating node state to member for oss1 Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/193, version=0.244.21): ok (rc=0) Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/lrm (origin=local/crmd/194, version=0.244.22): ok (rc=0) Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss2']/transient_attributes (origin=oss2/crmd/6, version=0.244.23): ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss1']/lrm": ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: do_dc_join_ack: join-9: Updating node state to member for mds2 Jan 4 10:13:32 mds1 crmd: [1305]: info: do_dc_join_ack: join-9: Updating node state to member for oss2 Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds2']/lrm (origin=local/crmd/196, version=0.244.24): ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: do_dc_join_ack: join-9: Updating node state to member for mds1 Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss2']/lrm (origin=local/crmd/198, version=0.244.25): ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='mds2']/lrm": ok (rc=0) Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds1']/lrm (origin=local/crmd/200, version=0.244.27): ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss2']/lrm": ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='mds1']/lrm": ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 10:13:32 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/202, version=0.244.28): ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date Jan 4 10:13:32 mds1 crmd: [1305]: info: crm_update_quorum: Updating quorum status to true (call=204) Jan 4 10:13:32 mds1 attrd: [1303]: info: attrd_local_callback: Sending full refresh (origin=crmd) Jan 4 10:13:32 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/204, version=0.244.28): ok (rc=0) Jan 4 10:13:32 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:176 - Triggered transition abort (complete=1) : Peer Cancelled Jan 4 10:13:32 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Jan 4 10:13:32 mds1 crmd: [1305]: info: do_pe_invoke: Query 205: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:13:32 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Jan 4 10:13:32 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=205, ref=pe_calc-dc-1325690012-174, seq=5020, quorate=1 Jan 4 10:13:32 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:13:32 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Jan 4 10:13:32 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:13:32 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:13:32 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:13:32 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:13:32 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:13:32 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:13:32 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:13:32 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:13:32 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:13:32 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:13:32 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:13:32 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:13:32 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:32 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:32 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:32 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:32 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:13:32 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:13:32 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:13:32 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:13:32 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:13:32 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:13:32 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:13:32 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:13:33 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 10:13:33 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 10:13:33 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:13:33 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:13:33 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:13:33 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:13:33 mds1 pengine: [1304]: notice: LogActions: Move resource st-mds1#011(Started mds1 -> oss1) Jan 4 10:13:33 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:13:33 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started oss1 -> mds1) Jan 4 10:13:33 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started oss1 -> mds1) Jan 4 10:13:33 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0001_4#011(Started mds2 -> oss1) Jan 4 10:13:33 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0000_3#011(Started mds2 -> oss1) Jan 4 10:13:33 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 10:13:33 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 88: 25 actions in 25 synapses Jan 4 10:13:33 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 88 (ref=pe_calc-dc-1325690012-174) derived from /var/lib/pengine/pe-input-28.bz2 Jan 4 10:13:33 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 5: monitor st-oss2_monitor_0 on oss2 Jan 4 10:13:33 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 6: monitor st-oss1_monitor_0 on oss2 Jan 4 10:13:33 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 7: monitor st-mds1_monitor_0 on oss2 Jan 4 10:13:33 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 8: monitor st-mds2_monitor_0 on oss2 Jan 4 10:13:33 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 9: monitor MGS_1_monitor_0 on oss2 Jan 4 10:13:33 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 10: monitor lustre-MDT0000_2_monitor_0 on oss2 Jan 4 10:13:33 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 11: monitor lustre-OST0001_4_monitor_0 on oss2 Jan 4 10:13:33 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 12: monitor lustre-OST0000_3_monitor_0 on oss2 Jan 4 10:13:33 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_monitor_0 (5) confirmed on oss2 (rc=0) Jan 4 10:13:33 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_monitor_0 (7) confirmed on oss2 (rc=0) Jan 4 10:13:33 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_monitor_0 (6) confirmed on oss2 (rc=0) Jan 4 10:13:33 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_monitor_0 (8) confirmed on oss2 (rc=0) Jan 4 10:13:33 mds1 pengine: [1304]: info: process_pe_message: Transition 88: PEngine Input stored in: /var/lib/pengine/pe-input-28.bz2 Jan 4 10:13:34 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (11) confirmed on oss2 (rc=0) Jan 4 10:13:34 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (12) confirmed on oss2 (rc=0) Jan 4 10:13:34 mds1 crmd: [1305]: WARN: status_from_rc: Action 10 (lustre-MDT0000_2_monitor_0) on oss2 failed (target: 7 vs. rc: 0): Error Jan 4 10:13:34 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_monitor_0, magic=0:0;10:88:7:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.244.35) : Event failed Jan 4 10:13:34 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Jan 4 10:13:34 mds1 crmd: [1305]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 10:13:34 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (10) confirmed on oss2 (rc=4) Jan 4 10:13:34 mds1 crmd: [1305]: WARN: status_from_rc: Action 9 (MGS_1_monitor_0) on oss2 failed (target: 7 vs. rc: 0): Error Jan 4 10:13:34 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_monitor_0, magic=0:0;9:88:7:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.244.36) : Event failed Jan 4 10:13:34 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_monitor_0 (9) confirmed on oss2 (rc=4) Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 4: probe_complete probe_complete on oss2 - no waiting Jan 4 10:13:34 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 10:13:34 mds1 crmd: [1305]: notice: run_graph: Transition 88 (Complete=9, Pending=0, Fired=0, Skipped=16, Incomplete=0, Source=/var/lib/pengine/pe-input-28.bz2): Stopped Jan 4 10:13:34 mds1 crmd: [1305]: info: te_graph_trigger: Transition 88 is now complete Jan 4 10:13:34 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 10:13:34 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 10:13:34 mds1 crmd: [1305]: info: do_pe_invoke: Query 206: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:13:34 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=206, ref=pe_calc-dc-1325690014-184, seq=5020, quorate=1 Jan 4 10:13:34 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:13:34 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:13:34 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:13:34 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:13:34 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:13:34 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:13:34 mds1 pengine: [1304]: ERROR: native_add_running: Resource ocf::Target:lustre-MDT0000_2 appears to be active on 2 nodes. Jan 4 10:13:34 mds1 pengine: [1304]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:13:34 mds1 pengine: [1304]: ERROR: native_add_running: Resource ocf::Target:MGS_1 appears to be active on 2 nodes. Jan 4 10:13:34 mds1 pengine: [1304]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target) Started Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: #0110 : oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: #0111 : oss2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target) Started Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: #0110 : oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: #0111 : oss2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:13:34 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:13:34 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:13:34 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:13:34 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:13:34 mds1 pengine: [1304]: WARN: native_create_actions: Attempting recovery of resource MGS_1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 10:13:34 mds1 pengine: [1304]: WARN: native_create_actions: Attempting recovery of resource lustre-MDT0000_2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource st-mds1#011(Started mds1 -> oss1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started oss1 -> mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started oss1 -> mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0001_4#011(Started mds2 -> oss1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0000_3#011(Started mds2 -> oss1) Jan 4 10:13:34 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss2, magic=NA, cib=0.244.37) : Transient attribute: update Jan 4 10:13:34 mds1 crmd: [1305]: info: do_pe_invoke: Query 207: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:13:34 mds1 crmd: [1305]: info: handle_response: pe_calc calculation pe_calc-dc-1325690014-184 is obsolete Jan 4 10:13:34 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=207, ref=pe_calc-dc-1325690014-185, seq=5020, quorate=1 Jan 4 10:13:34 mds1 pengine: [1304]: ERROR: process_pe_message: Transition 89: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-11.bz2 Jan 4 10:13:34 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:13:34 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:13:34 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:13:34 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:13:34 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:13:34 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:13:34 mds1 pengine: [1304]: ERROR: native_add_running: Resource ocf::Target:lustre-MDT0000_2 appears to be active on 2 nodes. Jan 4 10:13:34 mds1 pengine: [1304]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 10:13:34 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:13:34 mds1 pengine: [1304]: ERROR: native_add_running: Resource ocf::Target:MGS_1 appears to be active on 2 nodes. Jan 4 10:13:34 mds1 pengine: [1304]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target) Started Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: #0110 : oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: #0111 : oss2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target) Started Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: #0110 : oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: #0111 : oss2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:13:34 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:13:34 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:13:34 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:13:34 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:13:34 mds1 pengine: [1304]: WARN: native_create_actions: Attempting recovery of resource MGS_1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 10:13:34 mds1 pengine: [1304]: WARN: native_create_actions: Attempting recovery of resource lustre-MDT0000_2 Jan 4 10:13:34 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource st-mds1#011(Started mds1 -> oss1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started oss1 -> mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started oss1 -> mds1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0001_4#011(Started mds2 -> oss1) Jan 4 10:13:34 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0000_3#011(Started mds2 -> oss1) Jan 4 10:13:34 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 10:13:34 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 90: 17 actions in 17 synapses Jan 4 10:13:34 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 90 (ref=pe_calc-dc-1325690014-185) derived from /var/lib/pengine/pe-error-12.bz2 Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 11: stop st-mds1_stop_0 on mds1 (local) Jan 4 10:13:34 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=11:90:0:d3f0e147-de8d-45fc-bf26-c4214c060be6 op=st-mds1_stop_0 ) Jan 4 10:13:34 mds1 lrmd: [1302]: info: rsc:st-mds1:16: stop Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 15: stop MGS_1_stop_0 on oss1 Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 16: stop MGS_1_stop_0 on oss2 Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 19: stop lustre-MDT0000_2_stop_0 on oss1 Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 20: stop lustre-MDT0000_2_stop_0 on oss2 Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 23: stop lustre-OST0001_4_stop_0 on mds2 Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 26: stop lustre-OST0000_3_stop_0 on mds2 Jan 4 10:13:34 mds1 stonith-ng: [1300]: info: stonith_device_remove: Removed 'st-mds1' from the device list (3 active devices) Jan 4 10:13:34 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_device_remove from lrmd: rc=0 Jan 4 10:13:34 mds1 crmd: [1305]: info: process_lrm_event: LRM operation st-mds1_stop_0 (call=16, rc=0, cib-update=208, confirmed=true) ok Jan 4 10:13:34 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_stop_0 (11) confirmed on mds1 (rc=0) Jan 4 10:13:34 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 12: start st-mds1_start_0 on oss1 Jan 4 10:13:34 mds1 pengine: [1304]: ERROR: process_pe_message: Transition 90: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-12.bz2 Jan 4 10:13:34 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_start_0 (12) confirmed on oss1 (rc=0) Jan 4 10:13:35 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_stop_0 (20) confirmed on oss2 (rc=0) Jan 4 10:13:35 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_stop_0 (16) confirmed on oss2 (rc=0) Jan 4 10:13:35 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_stop_0 (19) confirmed on oss1 (rc=0) Jan 4 10:13:35 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 21: start lustre-MDT0000_2_start_0 on mds1 (local) Jan 4 10:13:35 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=21:90:0:d3f0e147-de8d-45fc-bf26-c4214c060be6 op=lustre-MDT0000_2_start_0 ) Jan 4 10:13:35 mds1 lrmd: [1302]: info: rsc:lustre-MDT0000_2:17: start Jan 4 10:13:35 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_stop_0 (15) confirmed on oss1 (rc=0) Jan 4 10:13:35 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 17: start MGS_1_start_0 on mds1 (local) Jan 4 10:13:35 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=17:90:0:d3f0e147-de8d-45fc-bf26-c4214c060be6 op=MGS_1_start_0 ) Jan 4 10:13:35 mds1 lrmd: [1302]: info: rsc:MGS_1:18: start Jan 4 10:13:35 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_stop_0 (23) confirmed on mds2 (rc=0) Jan 4 10:13:35 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 24: start lustre-OST0001_4_start_0 on oss1 Jan 4 10:13:35 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_stop_0 (26) confirmed on mds2 (rc=0) Jan 4 10:13:35 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 27: start lustre-OST0000_3_start_0 on oss1 Jan 4 10:13:35 mds1 crmd: [1305]: info: te_pseudo_action: Pseudo action 1 fired and confirmed Jan 4 10:13:37 mds1 kernel: Lustre: OBD class driver, http://wiki.whamcloud.com/ Jan 4 10:13:37 mds1 kernel: Lustre: Lustre Version: 2.1.0 Jan 4 10:13:37 mds1 kernel: Lustre: Build Version: RC2-g9d71fe8-PRISTINE-2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 Jan 4 10:13:37 mds1 kernel: Lustre: Lustre LU module (ffffffffa052bf60). Jan 4 10:13:41 mds1 corosync[1294]: [TOTEM ] A processor failed, forming new configuration. Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5024: memb=3, new=0, lost=1 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds2 2608507072 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: oss2 2692393152 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: lost: oss1 2474289344 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5024: memb=3, new=0, lost=0 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds2 2608507072 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: oss2 2692393152 Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: ais_mark_unseen_peer_dead: Node oss1 was not seen in the previous transition Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: update_member: Node 2474289344/oss1 is now: lost Jan 4 10:13:42 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 5024 to 2 children Jan 4 10:13:42 mds1 cib: [1301]: info: ais_dispatch_message: Membership 5024: quorum retained Jan 4 10:13:42 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5024: quorum retained Jan 4 10:13:42 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 10:13:42 mds1 cib: [1301]: info: crm_update_peer: Node oss1: id=2474289344 state=lost (new) addr=r(0) ip(192.168.122.147) votes=1 born=4672 seen=5020 proc=00000000000000000000000000111312 Jan 4 10:13:42 mds1 crmd: [1305]: info: ais_status_callback: status: oss1 is now lost (was member) Jan 4 10:13:42 mds1 crmd: [1305]: info: crm_update_peer: Node oss1: id=2474289344 state=lost (new) addr=r(0) ip(192.168.122.147) votes=1 born=4672 seen=5020 proc=00000000000000000000000000111312 Jan 4 10:13:42 mds1 crmd: [1305]: info: erase_node_from_join: Removed node oss1 from join calculations: welcomed=0 itegrated=0 finalized=0 confirmed=1 Jan 4 10:13:42 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/209, version=0.244.45): ok (rc=0) Jan 4 10:13:42 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:42 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/212, version=0.244.46): ok (rc=0) Jan 4 10:13:42 mds1 crmd: [1305]: WARN: match_down_event: No match for shutdown action on oss1 Jan 4 10:13:42 mds1 crmd: [1305]: info: te_update_diff: Stonith/shutdown of oss1 not matched Jan 4 10:13:42 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:194 - Triggered transition abort (complete=0, tag=node_state, id=oss1, magic=NA, cib=0.244.46) : Node failure Jan 4 10:13:42 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Jan 4 10:13:42 mds1 crmd: [1305]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 10:13:42 mds1 crmd: [1305]: notice: fail_incompletable_actions: Action 24 (24) is scheduled for oss1 (offline) Jan 4 10:13:42 mds1 crmd: [1305]: notice: fail_incompletable_actions: Action 25 (25) is scheduled for oss1 (offline) Jan 4 10:13:42 mds1 crmd: [1305]: notice: fail_incompletable_actions: Action 27 (27) is scheduled for oss1 (offline) Jan 4 10:13:42 mds1 crmd: [1305]: notice: fail_incompletable_actions: Action 28 (28) is scheduled for oss1 (offline) Jan 4 10:13:42 mds1 crmd: [1305]: WARN: fail_incompletable_actions: Node oss1 shutdown resulted in un-runnable actions Jan 4 10:13:42 mds1 crmd: [1305]: info: abort_transition_graph: fail_incompletable_actions:107 - Triggered transition abort (complete=0, tag=rsc_op, id=28, magic=NA) : Node failure Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5028: memb=3, new=0, lost=0 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: mds2 2608507072 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: memb: oss2 2692393152 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5028: memb=4, new=1, lost=0 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: update_member: Node 2474289344/oss1 is now: member Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: NEW: oss1 2474289344 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: oss1 2474289344 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: mds2 2608507072 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: pcmk_peer_update: MEMB: oss2 2692393152 Jan 4 10:13:43 mds1 corosync[1294]: [pcmk ] info: send_member_notification: Sending membership update 5028 to 2 children Jan 4 10:13:43 mds1 cib: [1301]: info: ais_dispatch_message: Membership 5028: quorum retained Jan 4 10:13:43 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5028: quorum retained Jan 4 10:13:43 mds1 corosync[1294]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 10:13:43 mds1 cib: [1301]: info: crm_update_peer: Node oss1: id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=1 born=4672 seen=5028 proc=00000000000000000000000000111312 Jan 4 10:13:43 mds1 crmd: [1305]: info: ais_status_callback: status: oss1 is now member (was lost) Jan 4 10:13:43 mds1 crmd: [1305]: info: crm_update_peer: Node oss1: id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=1 born=4672 seen=5028 proc=00000000000000000000000000111312 Jan 4 10:13:43 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/lrm (origin=local/crmd/213, version=0.244.47): ok (rc=0) Jan 4 10:13:43 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/transient_attributes (origin=local/crmd/214, version=0.244.48): ok (rc=0) Jan 4 10:13:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:13:43 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/215, version=0.244.48): ok (rc=0) Jan 4 10:13:43 mds1 cib: [1301]: info: cib_process_readwrite: We are now in R/O mode Jan 4 10:13:43 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:43 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=oss1/crmd/23, version=0.244.49): ok (rc=0) Jan 4 10:13:43 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:270 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_monitor_0, magic=0:0;9:87:7:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.244.47) : Resource op removal Jan 4 10:13:43 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/218, version=0.244.49): ok (rc=0) Jan 4 10:13:43 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss1']/lrm": ok (rc=0) Jan 4 10:13:43 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:157 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.244.48) : Transient attribute: removal Jan 4 10:13:43 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss1']/transient_attributes": ok (rc=0) Jan 4 10:13:43 mds1 cib: [1301]: WARN: cib_process_diff: Diff 0.244.45 -> 0.244.46 not applied to 0.244.49: current "num_updates" is greater than required Jan 4 10:13:43 mds1 cib: [1301]: WARN: cib_process_diff: Diff 0.244.46 -> 0.244.47 not applied to 0.244.49: current "num_updates" is greater than required Jan 4 10:13:43 mds1 cib: [1301]: WARN: cib_process_diff: Diff 0.244.47 -> 0.244.48 not applied to 0.244.49: current "num_updates" is greater than required Jan 4 10:13:43 mds1 crmd: [1305]: ERROR: crmd_ha_msg_filter: Another DC detected: oss1 (op=noop) Jan 4 10:13:43 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ] Jan 4 10:13:43 mds1 crmd: [1305]: info: update_dc: Unset DC mds1 Jan 4 10:13:43 mds1 cib: [1301]: WARN: cib_process_diff: Diff 0.244.48 -> 0.244.49 not applied to 0.244.49: current "num_updates" is greater than required Jan 4 10:13:43 mds1 cib: [1301]: info: apply_xml_diff: Digest mis-match: expected c3c8c3c054b0448e92f81e02b70362e5, calculated 4ce7bf14da36c3c5f635dbd475c20a0e Jan 4 10:13:43 mds1 cib: [1301]: notice: cib_process_diff: Diff 0.244.49 -> 0.244.50 not applied to 0.244.49: Failed application of an update diff Jan 4 10:13:43 mds1 cib: [1301]: info: cib_server_process_diff: Requesting re-sync from peer Jan 4 10:13:43 mds1 cib: [1301]: WARN: cib_diff_notify: Local-only Change (client:crmd, call: 37): -1.-1.-1 (Application of an update diff failed, requesting a full refresh) Jan 4 10:13:43 mds1 cib: [1301]: WARN: cib_server_process_diff: Not applying diff 0.244.50 -> 0.244.51 (sync in progress) Jan 4 10:13:43 mds1 corosync[1294]: [CPG ] downlist received left_list: 1 Jan 4 10:13:43 mds1 corosync[1294]: [CPG ] downlist received left_list: 1 Jan 4 10:13:43 mds1 corosync[1294]: [CPG ] downlist received left_list: 1 Jan 4 10:13:43 mds1 corosync[1294]: [CPG ] downlist received left_list: 0 Jan 4 10:13:43 mds1 corosync[1294]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 10:13:43 mds1 corosync[1294]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 10:13:43 mds1 cib: [1301]: WARN: cib_server_process_diff: Not applying diff 0.244.51 -> 0.245.1 (sync in progress) Jan 4 10:13:43 mds1 crmd: [1305]: info: do_election_count_vote: Election 3 (owner: oss1) pass: vote from oss1 (Uptime) Jan 4 10:13:43 mds1 crmd: [1305]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Jan 4 10:13:43 mds1 crmd: [1305]: info: do_dc_takeover: Taking over DC status for this partition Jan 4 10:13:43 mds1 cib: [1301]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-77.raw Jan 4 10:13:43 mds1 cib: [1301]: info: write_cib_contents: Wrote version 0.245.0 of the CIB to disk (digest: dd13e62d74d18d38d3467d20ec43aa1f) Jan 4 10:13:44 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.Q13cdF (digest: /var/lib/heartbeat/crm/cib.EwcsRE) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_replace_notify: Local-only Replace: -1.-1.-1 from oss1 Jan 4 10:13:44 mds1 attrd: [1303]: info: do_cib_replaced: Sending full refresh Jan 4 10:13:44 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_readwrite: We are now in R/W mode Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/220, version=0.245.1): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/221, version=0.245.1): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/223, version=0.245.2): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: join_make_offer: Making join offers based on membership 5028 Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/225, version=0.245.2): ok (rc=0) Jan 4 10:13:44 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Jan 4 10:13:44 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Jan 4 10:13:44 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-10: Waiting on 4 outstanding join acks Jan 4 10:13:44 mds1 crmd: [1305]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=0) : Non-status change Jan 4 10:13:44 mds1 crmd: [1305]: info: need_abort: Aborting on change to epoch Jan 4 10:13:44 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss2, magic=NA, cib=0.245.2) : Transient attribute: update Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/227, version=0.245.4): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.245.3) : Transient attribute: update Jan 4 10:13:44 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.245.4) : Transient attribute: update Jan 4 10:13:44 mds1 crmd: [1305]: info: config_query_callback: Checking for expired actions every 900000ms Jan 4 10:13:44 mds1 crmd: [1305]: info: config_query_callback: Sending expected-votes=4 to corosync Jan 4 10:13:44 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5028: quorum retained Jan 4 10:13:44 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/230, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: do_state_transition: State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=do_cib_replaced ] Jan 4 10:13:44 mds1 crmd: [1305]: WARN: do_log: FSA: Input I_JOIN_OFFER from route_message() received in state S_ELECTION Jan 4 10:13:44 mds1 crmd: [1305]: WARN: do_log: FSA: Input I_JOIN_REQUEST from route_message() received in state S_ELECTION Jan 4 10:13:44 mds1 crmd: [1305]: WARN: do_log: FSA: Input I_JOIN_REQUEST from route_message() received in state S_ELECTION Jan 4 10:13:44 mds1 crmd: [1305]: WARN: do_log: FSA: Input I_JOIN_REQUEST from route_message() received in state S_ELECTION Jan 4 10:13:44 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5028: quorum retained Jan 4 10:13:44 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/232, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Jan 4 10:13:44 mds1 crmd: [1305]: info: do_dc_takeover: Taking over DC status for this partition Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_readwrite: We are now in R/O mode Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_slave_all for section 'all' (origin=local/crmd/233, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_readwrite: We are now in R/W mode Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/234, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/235, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/237, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/239, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-11: Waiting on 4 outstanding join acks Jan 4 10:13:44 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5028: quorum retained Jan 4 10:13:44 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/242, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: config_query_callback: Checking for expired actions every 900000ms Jan 4 10:13:44 mds1 crmd: [1305]: info: config_query_callback: Sending expected-votes=4 to corosync Jan 4 10:13:44 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 10:13:44 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5028: quorum retained Jan 4 10:13:44 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/245, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 10:13:44 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes responded to the join offer. Jan 4 10:13:44 mds1 crmd: [1305]: info: do_dc_join_finalize: join-11: Syncing the CIB from mds1 to the rest of the cluster Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/246, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/247, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/248, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/249, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/250, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: do_dc_join_ack: join-11: Updating node state to member for mds2 Jan 4 10:13:44 mds1 crmd: [1305]: info: do_dc_join_ack: join-11: Updating node state to member for oss2 Jan 4 10:13:44 mds1 crmd: [1305]: WARN: msg_to_op(1324): failed to get the value of field lrm_opstatus from a ha_msg Jan 4 10:13:44 mds1 crmd: [1305]: info: msg_to_op: Message follows: Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG: Dumping message with 16 fields Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[0] : [lrm_t=op] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[1] : [lrm_rid=lustre-MDT0000_2] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[2] : [lrm_op=start] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[3] : [lrm_timeout=300000] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[4] : [lrm_interval=0] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[5] : [lrm_delay=0] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[6] : [lrm_copyparams=1] Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds2']/lrm (origin=local/crmd/251, version=0.245.5): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[7] : [lrm_t_run=0] Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss2']/lrm (origin=local/crmd/253, version=0.245.6): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[8] : [lrm_t_rcchange=0] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[9] : [lrm_exec_time=0] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[10] : [lrm_queue_time=0] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[11] : [lrm_targetrc=-1] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[12] : [lrm_app=crmd] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[13] : [lrm_userdata=21:90:0:d3f0e147-de8d-45fc-bf26-c4214c060be6] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[14] : [(2)lrm_param=0x1ae5470(133 155)] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG: Dumping message with 4 fields Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[0] : [CRM_meta_name=start] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[1] : [crm_feature_set=3.0.2] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[2] : [target=c4f08837-0cea-46c1-a5d8-4ef412fadd0b] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[3] : [CRM_meta_timeout=300000] Jan 4 10:13:44 mds1 crmd: [1305]: info: MSG[15] : [lrm_callid=17] Jan 4 10:13:44 mds1 lrmd: [1302]: info: stonith_api_device_metadata: looking up fence_virsh/redhat metadata Jan 4 10:13:44 mds1 lrmd: [1302]: info: stonithRA plugin: got metadata: #012#012fence_virsh is an I/O Fencing agent which can be used with the virtual machines managed by libvirt. It logs via ssh to a dom0 and there run virsh command, which does all work. #012.P#012By default, virsh needs root account to do properly work. So you must allow ssh login in your sshd_config.#012#012#011#012#011#011#012#011#011#012#011#011Fencing Action#012#011#012#011#012#011#011#012#011#011#012#011#011IP Address or Hostname#012#011#012#011#012#011#011#012#011#011#012#011#011Login Name#012#011#012#011#012#011#011#012#011#011#012#011#011Login password or passphrase#012#011#012#011#012#011#011#012#011#011#012#011#011Script to retrieve password#012#011#012#011#012#011#011#012#011#011#012#011#011SSH connection#012#011#012#011#012#011#011 S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 10:13:44 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 10:13:44 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/259, version=0.245.10): ok (rc=0) Jan 4 10:13:44 mds1 crmd: [1305]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date Jan 4 10:13:44 mds1 crmd: [1305]: info: crm_update_quorum: Updating quorum status to true (call=261) Jan 4 10:13:44 mds1 attrd: [1303]: info: attrd_local_callback: Sending full refresh (origin=crmd) Jan 4 10:13:44 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:176 - Triggered transition abort (complete=0) : Peer Cancelled Jan 4 10:13:44 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Jan 4 10:13:44 mds1 crmd: [1305]: WARN: register_fsa_input_adv: do_te_invoke stalled the FSA with pending inputs Jan 4 10:13:45 mds1 cib: [1301]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-78.raw Jan 4 10:13:45 mds1 cib: [1301]: info: write_cib_contents: Wrote version 0.246.0 of the CIB to disk (digest: 24af8926d0fa3a13d10c930d04f509ad) Jan 4 10:13:45 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.etHrVG (digest: /var/lib/heartbeat/crm/cib.RxnrZI) Jan 4 10:13:45 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 10:13:45 mds1 cib: [1301]: info: log_data_element: cib:diff: + Jan 4 10:13:45 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/261, version=0.246.1): ok (rc=0) Jan 4 10:13:45 mds1 crmd: [1305]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=0) : Non-status change Jan 4 10:13:45 mds1 crmd: [1305]: info: need_abort: Aborting on change to admin_epoch Jan 4 10:13:45 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Jan 4 10:13:45 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Jan 4 10:14:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:14:08 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:14:08 mds1 kernel: fld: Unknown symbol RQF_FLD_QUERY Jan 4 10:14:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:14:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:14:38 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:14:38 mds1 kernel: fld: Unknown symbol req_capsule_server_pack Jan 4 10:14:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:15:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:15:08 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:15:08 mds1 kernel: fld: Unknown symbol req_capsule_client_get Jan 4 10:15:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:15:38 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:15:38 mds1 kernel: fld: Unknown symbol ptlrpc_queue_wait Jan 4 10:15:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:15:54 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 10:15:54 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 10:15:54 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1363 seconds. Jan 4 10:16:08 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:16:08 mds1 kernel: fld: Unknown symbol req_capsule_fini Jan 4 10:16:38 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:16:38 mds1 kernel: fld: Unknown symbol req_capsule_init Jan 4 10:17:01 mds1 kernel: INFO: task modprobe:4047 blocked for more than 120 seconds. Jan 4 10:17:01 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 4 10:17:01 mds1 kernel: modprobe D 0000000000000000 0 4047 4040 0x00000080 Jan 4 10:17:01 mds1 kernel: ffff88002e8a9aa8 0000000000000082 0000000000000000 ffff88003e7260f8 Jan 4 10:17:01 mds1 kernel: ffff880002015f80 ffff88003e7260c0 ffff880002015f80 0000000000000000 Jan 4 10:17:01 mds1 kernel: ffff880037c39af8 ffff88002e8a9fd8 000000000000f598 ffff880037c39af8 Jan 4 10:17:01 mds1 kernel: Call Trace: Jan 4 10:17:01 mds1 kernel: [] schedule_timeout+0x215/0x2e0 Jan 4 10:17:01 mds1 kernel: [] ? autoremove_wake_function+0x16/0x40 Jan 4 10:17:01 mds1 kernel: [] ? __wake_up_common+0x59/0x90 Jan 4 10:17:01 mds1 kernel: [] wait_for_common+0x123/0x180 Jan 4 10:17:01 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 10:17:01 mds1 kernel: [] ? __queue_work+0x41/0x50 Jan 4 10:17:01 mds1 kernel: [] wait_for_completion+0x1d/0x20 Jan 4 10:17:01 mds1 kernel: [] call_usermodehelper_exec+0xe0/0xf0 Jan 4 10:17:01 mds1 kernel: [] ? lnet_startup_lndnis+0xe6/0x6e0 [lnet] Jan 4 10:17:01 mds1 kernel: [] __request_module+0x18b/0x210 Jan 4 10:17:01 mds1 kernel: [] ? lnet_parse_networks+0xb0/0x7e0 [lnet] Jan 4 10:17:01 mds1 kernel: [] ? pvclock_clocksource_read+0x58/0xd0 Jan 4 10:17:01 mds1 kernel: [] lnet_startup_lndnis+0xe6/0x6e0 [lnet] Jan 4 10:17:01 mds1 kernel: [] ? cfs_alloc+0x63/0x90 [libcfs] Jan 4 10:17:01 mds1 kernel: [] LNetNIInit+0x125/0x1f0 [lnet] Jan 4 10:17:01 mds1 kernel: [] ? __cond_resched+0x2a/0x40 Jan 4 10:17:01 mds1 kernel: [] ? init_module+0x0/0x597 [ptlrpc] Jan 4 10:17:01 mds1 kernel: [] ptlrpc_ni_init+0x29/0x170 [ptlrpc] Jan 4 10:17:01 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 10:17:01 mds1 kernel: [] ptlrpc_init_portals+0x13/0xd0 [ptlrpc] Jan 4 10:17:01 mds1 kernel: [] ? init_module+0x0/0x597 [ptlrpc] Jan 4 10:17:01 mds1 kernel: [] init_module+0xe0/0x597 [ptlrpc] Jan 4 10:17:01 mds1 kernel: [] ? __blocking_notifier_call_chain+0x65/0x80 Jan 4 10:17:01 mds1 kernel: [] do_one_initcall+0x3c/0x1d0 Jan 4 10:17:01 mds1 kernel: [] sys_init_module+0xdf/0x250 Jan 4 10:17:01 mds1 kernel: [] system_call_fastpath+0x16/0x1b Jan 4 10:17:08 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:17:08 mds1 kernel: fld: Unknown symbol req_capsule_set Jan 4 10:17:38 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:17:38 mds1 kernel: fld: Unknown symbol req_capsule_server_get Jan 4 10:18:08 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:18:08 mds1 kernel: fld: Unknown symbol ptlrpc_at_set_req_timeout Jan 4 10:18:35 mds1 lrmd: [1302]: WARN: lustre-MDT0000_2:start process (PID 4039) timed out (try 1). Killing with signal SIGTERM (15). Jan 4 10:18:35 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:18:35 mds1 kernel: fld: Unknown symbol ptlrpc_request_alloc_pack Jan 4 10:18:35 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:18:35 mds1 kernel: fld: Unknown symbol RMF_FLD_OPC Jan 4 10:18:35 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:18:35 mds1 kernel: fld: Unknown symbol ptlrpc_request_set_replen Jan 4 10:18:35 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:18:35 mds1 kernel: fld: Unknown symbol RMF_FLD_MDFLD Jan 4 10:18:35 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 10:18:35 mds1 kernel: fld: Unknown symbol ptlrpc_req_finished Jan 4 10:18:35 mds1 lrmd: [1302]: WARN: MGS_1:start process (PID 4040) timed out (try 1). Killing with signal SIGTERM (15). Jan 4 10:18:35 mds1 lrmd: [1302]: WARN: operation start[18] on ocf::Target::MGS_1 for client 1305, its parameters: CRM_meta_name=[start] crm_feature_set=[3.0.2] target=[96c8dccb-368b-4ffc-af7e-6b95cdcacadb] CRM_meta_timeout=[300000] : pid [4040] timed out Jan 4 10:18:35 mds1 crmd: [1305]: ERROR: process_lrm_event: LRM operation MGS_1_start_0 (18) Timed Out (timeout=300000ms) Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:176 - Triggered transition abort (complete=0) : Peer Cancelled Jan 4 10:18:35 mds1 crmd: [1305]: WARN: status_from_rc: Action 17 (MGS_1_start_0) on mds1 failed (target: 0 vs. rc: -2): Error Jan 4 10:18:35 mds1 crmd: [1305]: WARN: update_failcount: Updating failcount for MGS_1 on mds1 after failed start: rc=-2 (update=INFINITY, time=1325690315) Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_start_0, magic=2:-2;17:90:0:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.246.2) : Event failed Jan 4 10:18:35 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_start_0 (17) confirmed on mds1 (rc=4) Jan 4 10:18:35 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for fail-count-MGS_1 Jan 4 10:18:35 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-MGS_1 (INFINITY) Jan 4 10:18:35 mds1 attrd: [1303]: info: attrd_perform_update: Sent update 96: fail-count-MGS_1=INFINITY Jan 4 10:18:35 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for last-failure-MGS_1 Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.246.3) : Transient attribute: update Jan 4 10:18:35 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-MGS_1 (1325690315) Jan 4 10:18:35 mds1 attrd: [1303]: info: attrd_perform_update: Sent update 99: last-failure-MGS_1=1325690315 Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.246.4) : Transient attribute: update Jan 4 10:18:35 mds1 lrmd: [1302]: WARN: operation start[17] on ocf::Target::lustre-MDT0000_2 for client 1305, its parameters: CRM_meta_name=[start] crm_feature_set=[3.0.2] target=[c4f08837-0cea-46c1-a5d8-4ef412fadd0b] CRM_meta_timeout=[300000] : pid [4039] timed out Jan 4 10:18:35 mds1 crmd: [1305]: ERROR: process_lrm_event: LRM operation lustre-MDT0000_2_start_0 (17) Timed Out (timeout=300000ms) Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:176 - Triggered transition abort (complete=0) : Peer Cancelled Jan 4 10:18:35 mds1 crmd: [1305]: WARN: status_from_rc: Action 21 (lustre-MDT0000_2_start_0) on mds1 failed (target: 0 vs. rc: -2): Error Jan 4 10:18:35 mds1 crmd: [1305]: WARN: update_failcount: Updating failcount for lustre-MDT0000_2 on mds1 after failed start: rc=-2 (update=INFINITY, time=1325690315) Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_start_0, magic=2:-2;21:90:0:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.246.5) : Event failed Jan 4 10:18:35 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_start_0 (21) confirmed on mds1 (rc=4) Jan 4 10:18:35 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 10:18:35 mds1 crmd: [1305]: notice: run_graph: Transition 90 (Complete=13, Pending=0, Fired=0, Skipped=4, Incomplete=0, Source=/var/lib/pengine/pe-error-12.bz2): Stopped Jan 4 10:18:35 mds1 crmd: [1305]: info: te_graph_trigger: Transition 90 is now complete Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:176 - Triggered transition abort (complete=1) : Peer Cancelled Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 264: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:35 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for fail-count-lustre-MDT0000_2 Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 265: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:35 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-lustre-MDT0000_2 (INFINITY) Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=265, ref=pe_calc-dc-1325690315-218, seq=5028, quorate=1 Jan 4 10:18:35 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:18:35 mds1 attrd: [1303]: info: attrd_perform_update: Sent update 102: fail-count-lustre-MDT0000_2=INFINITY Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=mds1, magic=NA, cib=0.246.6) : Transient attribute: update Jan 4 10:18:35 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:18:35 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for last-failure-lustre-MDT0000_2 Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 266: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:35 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:18:35 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-lustre-MDT0000_2 (1325690315) Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=266, ref=pe_calc-dc-1325690315-219, seq=5028, quorate=1 Jan 4 10:18:35 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:18:35 mds1 attrd: [1303]: info: attrd_perform_update: Sent update 105: last-failure-lustre-MDT0000_2=1325690315 Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=mds1, magic=NA, cib=0.246.7) : Transient attribute: update Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 267: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=267, ref=pe_calc-dc-1325690315-220, seq=5028, quorate=1 Jan 4 10:18:35 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:18:35 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:18:35 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:18:35 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:35 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:18:35 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:18:35 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:18:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:18:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:18:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:18:35 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Recover resource lustre-MDT0000_2#011(Started mds1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Restart resource lustre-OST0001_4#011(Started oss1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Restart resource lustre-OST0000_3#011(Started oss1) Jan 4 10:18:35 mds1 crmd: [1305]: info: handle_response: pe_calc calculation pe_calc-dc-1325690315-218 is obsolete Jan 4 10:18:35 mds1 kernel: LustreError: 4047:0:(socklnd.c:2420:ksocknal_base_startup()) Can't spawn socknal scheduler[0]: -513 Jan 4 10:18:35 mds1 kernel: LustreError: 105-4: Error -100 starting up LNI tcp Jan 4 10:18:35 mds1 kernel: LustreError: 4047:0:(events.c:728:ptlrpc_init_portals()) network initialisation failed Jan 4 10:18:35 mds1 pengine: [1304]: info: process_pe_message: Transition 91: PEngine Input stored in: /var/lib/pengine/pe-input-29.bz2 Jan 4 10:18:35 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:18:35 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:18:35 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:18:35 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:18:35 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:18:35 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:18:35 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:18:35 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:35 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:18:35 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:18:35 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:35 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:18:35 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:18:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:18:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:18:35 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:18:35 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds2 Jan 4 10:18:35 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds1 -> mds2) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Restart resource lustre-OST0001_4#011(Started oss1) Jan 4 10:18:35 mds1 pengine: [1304]: notice: LogActions: Restart resource lustre-OST0000_3#011(Started oss1) Jan 4 10:18:35 mds1 crmd: [1305]: info: handle_response: pe_calc calculation pe_calc-dc-1325690315-219 is obsolete Jan 4 10:18:35 mds1 crmd: [1305]: info: process_graph_event: Action lustre-OST0001_4_start_0 arrived after a completed transition Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: process_graph_event:472 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=lustre-OST0001_4_start_0, magic=2:-2;24:90:0:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.246.8) : Inactive graph Jan 4 10:18:35 mds1 crmd: [1305]: WARN: update_failcount: Updating failcount for lustre-OST0001_4 on oss1 after failed start: rc=-2 (update=INFINITY, time=1325690315) Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 268: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=268, ref=pe_calc-dc-1325690315-221, seq=5028, quorate=1 Jan 4 10:18:35 mds1 crmd: [1305]: info: process_graph_event: Action lustre-OST0000_3_start_0 arrived after a completed transition Jan 4 10:18:35 mds1 crmd: [1305]: info: abort_transition_graph: process_graph_event:472 - Triggered transition abort (complete=1, tag=lrm_rsc_op, id=lustre-OST0000_3_start_0, magic=2:-2;27:90:0:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.246.9) : Inactive graph Jan 4 10:18:35 mds1 crmd: [1305]: WARN: update_failcount: Updating failcount for lustre-OST0000_3 on oss1 after failed start: rc=-2 (update=INFINITY, time=1325690315) Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke: Query 269: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:35 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=269, ref=pe_calc-dc-1325690315-222, seq=5028, quorate=1 Jan 4 10:18:36 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for fail-count-lustre-OST0001_4 Jan 4 10:18:36 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss1, magic=NA, cib=0.246.10) : Transient attribute: update Jan 4 10:18:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 270: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=270, ref=pe_calc-dc-1325690316-223, seq=5028, quorate=1 Jan 4 10:18:36 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for last-failure-lustre-OST0001_4 Jan 4 10:18:36 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss1, magic=NA, cib=0.246.11) : Transient attribute: update Jan 4 10:18:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 271: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:36 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss1, magic=NA, cib=0.246.12) : Transient attribute: update Jan 4 10:18:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 272: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:36 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss1, magic=NA, cib=0.246.13) : Transient attribute: update Jan 4 10:18:36 mds1 crmd: [1305]: info: do_pe_invoke: Query 273: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:18:36 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for fail-count-lustre-OST0000_3 Jan 4 10:18:36 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=273, ref=pe_calc-dc-1325690316-224, seq=5028, quorate=1 Jan 4 10:18:36 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for last-failure-lustre-OST0000_3 Jan 4 10:18:36 mds1 pengine: [1304]: info: process_pe_message: Transition 92: PEngine Input stored in: /var/lib/pengine/pe-input-30.bz2 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Restart resource lustre-OST0001_4#011(Started oss1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Restart resource lustre-OST0000_3#011(Started oss1) Jan 4 10:18:36 mds1 crmd: [1305]: info: handle_response: pe_calc calculation pe_calc-dc-1325690315-220 is obsolete Jan 4 10:18:36 mds1 pengine: [1304]: info: process_pe_message: Transition 93: PEngine Input stored in: /var/lib/pengine/pe-input-31.bz2 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Recover resource lustre-OST0001_4#011(Started oss1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Restart resource lustre-OST0000_3#011(Started oss1) Jan 4 10:18:36 mds1 crmd: [1305]: info: handle_response: pe_calc calculation pe_calc-dc-1325690315-221 is obsolete Jan 4 10:18:36 mds1 pengine: [1304]: info: process_pe_message: Transition 94: PEngine Input stored in: /var/lib/pengine/pe-input-32.bz2 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Recover resource lustre-OST0001_4#011(Started oss1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Recover resource lustre-OST0000_3#011(Started oss1) Jan 4 10:18:36 mds1 crmd: [1305]: info: handle_response: pe_calc calculation pe_calc-dc-1325690315-222 is obsolete Jan 4 10:18:36 mds1 pengine: [1304]: info: process_pe_message: Transition 95: PEngine Input stored in: /var/lib/pengine/pe-input-33.bz2 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0001_4#011(Started oss1 -> oss2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Recover resource lustre-OST0000_3#011(Started oss1) Jan 4 10:18:36 mds1 crmd: [1305]: info: handle_response: pe_calc calculation pe_calc-dc-1325690316-223 is obsolete Jan 4 10:18:36 mds1 pengine: [1304]: info: process_pe_message: Transition 96: PEngine Input stored in: /var/lib/pengine/pe-input-34.bz2 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:18:36 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:18:36 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:18:36 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:18:36 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss2 Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds1 -> mds2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0001_4#011(Started oss1 -> oss2) Jan 4 10:18:36 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0000_3#011(Started oss1 -> oss2) Jan 4 10:18:36 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 10:18:36 mds1 crmd: [1305]: WARN: destroy_action: Cancelling timer for action 24 (src=324) Jan 4 10:18:36 mds1 crmd: [1305]: WARN: destroy_action: Cancelling timer for action 27 (src=325) Jan 4 10:18:36 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 97: 13 actions in 13 synapses Jan 4 10:18:36 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 97 (ref=pe_calc-dc-1325690316-224) derived from /var/lib/pengine/pe-input-35.bz2 Jan 4 10:18:36 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 4: stop MGS_1_stop_0 on mds1 (local) Jan 4 10:18:36 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=4:97:0:d3f0e147-de8d-45fc-bf26-c4214c060be6 op=MGS_1_stop_0 ) Jan 4 10:18:36 mds1 lrmd: [1302]: info: rsc:MGS_1:19: stop Jan 4 10:18:36 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 3: stop lustre-MDT0000_2_stop_0 on mds1 (local) Jan 4 10:18:36 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=3:97:0:d3f0e147-de8d-45fc-bf26-c4214c060be6 op=lustre-MDT0000_2_stop_0 ) Jan 4 10:18:36 mds1 lrmd: [1302]: info: rsc:lustre-MDT0000_2:20: stop Jan 4 10:18:36 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 2: stop lustre-OST0001_4_stop_0 on oss1 Jan 4 10:18:36 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 1: stop lustre-OST0000_3_stop_0 on oss1 Jan 4 10:18:36 mds1 pengine: [1304]: info: process_pe_message: Transition 97: PEngine Input stored in: /var/lib/pengine/pe-input-35.bz2 Jan 4 10:18:36 mds1 crmd: [1305]: info: process_lrm_event: LRM operation MGS_1_stop_0 (call=19, rc=0, cib-update=274, confirmed=true) ok Jan 4 10:18:36 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_stop_0 (4) confirmed on mds1 (rc=0) Jan 4 10:18:36 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 19: start MGS_1_start_0 on mds2 Jan 4 10:18:36 mds1 crmd: [1305]: info: process_lrm_event: LRM operation lustre-MDT0000_2_stop_0 (call=20, rc=0, cib-update=275, confirmed=true) ok Jan 4 10:18:36 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_stop_0 (3) confirmed on mds1 (rc=0) Jan 4 10:18:36 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 21: start lustre-MDT0000_2_start_0 on mds2 Jan 4 10:18:36 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_stop_0 (1) confirmed on oss1 (rc=0) Jan 4 10:18:36 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 25: start lustre-OST0000_3_start_0 on oss2 Jan 4 10:18:36 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_stop_0 (2) confirmed on oss1 (rc=0) Jan 4 10:18:36 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 23: start lustre-OST0001_4_start_0 on oss2 Jan 4 10:18:36 mds1 crmd: [1305]: info: te_pseudo_action: Pseudo action 5 fired and confirmed Jan 4 10:21:01 mds1 cib: [1301]: info: cib_stats: Processed 343 operations (11807.00us average, 0% utilization) in the last 10min Jan 4 10:23:36 mds1 crmd: [1305]: WARN: status_from_rc: Action 19 (MGS_1_start_0) on mds2 failed (target: 0 vs. rc: -2): Error Jan 4 10:23:36 mds1 crmd: [1305]: WARN: update_failcount: Updating failcount for MGS_1 on mds2 after failed start: rc=-2 (update=INFINITY, time=1325690616) Jan 4 10:23:36 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_start_0, magic=2:-2;19:97:0:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.246.18) : Event failed Jan 4 10:23:36 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Jan 4 10:23:36 mds1 crmd: [1305]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 10:23:36 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_start_0 (19) confirmed on mds2 (rc=4) Jan 4 10:23:36 mds1 crmd: [1305]: WARN: status_from_rc: Action 21 (lustre-MDT0000_2_start_0) on mds2 failed (target: 0 vs. rc: -2): Error Jan 4 10:23:36 mds1 crmd: [1305]: WARN: update_failcount: Updating failcount for lustre-MDT0000_2 on mds2 after failed start: rc=-2 (update=INFINITY, time=1325690616) Jan 4 10:23:36 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_start_0, magic=2:-2;21:97:0:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.246.19) : Event failed Jan 4 10:23:36 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_start_0 (21) confirmed on mds2 (rc=4) Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.246.20) : Transient attribute: update Jan 4 10:23:37 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 1 to 1000000 Jan 4 10:23:37 mds1 crmd: [1305]: info: update_abort_priority: 'Event failed' abort superceeded Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.246.21) : Transient attribute: update Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.246.22) : Transient attribute: update Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.246.23) : Transient attribute: update Jan 4 10:23:37 mds1 crmd: [1305]: WARN: status_from_rc: Action 25 (lustre-OST0000_3_start_0) on oss2 failed (target: 0 vs. rc: -2): Error Jan 4 10:23:37 mds1 crmd: [1305]: WARN: update_failcount: Updating failcount for lustre-OST0000_3 on oss2 after failed start: rc=-2 (update=INFINITY, time=1325690617) Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0000_3_start_0, magic=2:-2;25:97:0:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.246.24) : Event failed Jan 4 10:23:37 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_start_0 (25) confirmed on oss2 (rc=4) Jan 4 10:23:37 mds1 crmd: [1305]: WARN: status_from_rc: Action 23 (lustre-OST0001_4_start_0) on oss2 failed (target: 0 vs. rc: -2): Error Jan 4 10:23:37 mds1 crmd: [1305]: WARN: update_failcount: Updating failcount for lustre-OST0001_4 on oss2 after failed start: rc=-2 (update=INFINITY, time=1325690617) Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0001_4_start_0, magic=2:-2;23:97:0:d3f0e147-de8d-45fc-bf26-c4214c060be6, cib=0.246.25) : Event failed Jan 4 10:23:37 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_start_0 (23) confirmed on oss2 (rc=4) Jan 4 10:23:37 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 10:23:37 mds1 crmd: [1305]: notice: run_graph: Transition 97 (Complete=9, Pending=0, Fired=0, Skipped=4, Incomplete=0, Source=/var/lib/pengine/pe-input-35.bz2): Stopped Jan 4 10:23:37 mds1 crmd: [1305]: info: te_graph_trigger: Transition 97 is now complete Jan 4 10:23:37 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 10:23:37 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 10:23:37 mds1 crmd: [1305]: info: do_pe_invoke: Query 276: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:23:37 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=276, ref=pe_calc-dc-1325690617-233, seq=5028, quorate=1 Jan 4 10:23:37 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss2, magic=NA, cib=0.246.26) : Transient attribute: update Jan 4 10:23:37 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss2, magic=NA, cib=0.246.27) : Transient attribute: update Jan 4 10:23:37 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss2, magic=NA, cib=0.246.28) : Transient attribute: update Jan 4 10:23:37 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:23:37 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss2, magic=NA, cib=0.246.29) : Transient attribute: update Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:23:37 mds1 crmd: [1305]: info: do_pe_invoke: Query 277: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 10:23:37 mds1 crmd: [1305]: info: do_pe_invoke: Query 278: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 10:23:37 mds1 crmd: [1305]: info: do_pe_invoke: Query 279: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:23:37 mds1 crmd: [1305]: info: do_pe_invoke: Query 280: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:23:37 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:23:37 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=280, ref=pe_calc-dc-1325690617-234, seq=5028, quorate=1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds2: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds2: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss2: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss2: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds2 FAILED Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds2 FAILED Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss2 FAILED Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss2 FAILED Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds2 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds2 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:23:37 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:23:37 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:23:37 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:23:37 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on oss1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds2 -> oss2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds2 -> oss1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Recover resource lustre-OST0001_4#011(Started oss2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Recover resource lustre-OST0000_3#011(Started oss2) Jan 4 10:23:37 mds1 crmd: [1305]: info: handle_response: pe_calc calculation pe_calc-dc-1325690617-233 is obsolete Jan 4 10:23:37 mds1 pengine: [1304]: info: process_pe_message: Transition 98: PEngine Input stored in: /var/lib/pengine/pe-input-36.bz2 Jan 4 10:23:37 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:23:37 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:23:37 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:23:37 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:23:37 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds2: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds2: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss2: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss2: unknown exec error (-2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds2 FAILED Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds2 FAILED Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss2 FAILED Jan 4 10:23:37 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss2 FAILED Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss2 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss2 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds2 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds2 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:23:37 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:23:37 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:23:37 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:23:37 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:23:37 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:23:37 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on oss1 Jan 4 10:23:37 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on mds2 Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started mds2 -> oss2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds2 -> oss1) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0001_4#011(Started oss2 -> mds2) Jan 4 10:23:37 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0000_3#011(Started oss2 -> mds2) Jan 4 10:23:37 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 10:23:37 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 99: 13 actions in 13 synapses Jan 4 10:23:37 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 99 (ref=pe_calc-dc-1325690617-234) derived from /var/lib/pengine/pe-input-37.bz2 Jan 4 10:23:37 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 2: stop MGS_1_stop_0 on mds2 Jan 4 10:23:37 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 1: stop lustre-MDT0000_2_stop_0 on mds2 Jan 4 10:23:37 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 4: stop lustre-OST0001_4_stop_0 on oss2 Jan 4 10:23:37 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 3: stop lustre-OST0000_3_stop_0 on oss2 Jan 4 10:23:37 mds1 pengine: [1304]: info: process_pe_message: Transition 99: PEngine Input stored in: /var/lib/pengine/pe-input-37.bz2 Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_stop_0 (4) confirmed on oss2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 23: start lustre-OST0001_4_start_0 on mds2 Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_stop_0 (3) confirmed on oss2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 25: start lustre-OST0000_3_start_0 on mds2 Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_stop_0 (1) confirmed on mds2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 21: start lustre-MDT0000_2_start_0 on oss1 Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_stop_0 (2) confirmed on mds2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 19: start MGS_1_start_0 on oss2 Jan 4 10:23:38 mds1 crmd: [1305]: info: te_pseudo_action: Pseudo action 5 fired and confirmed Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_start_0 (21) confirmed on oss1 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 22: monitor lustre-MDT0000_2_monitor_120000 on oss1 Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_start_0 (19) confirmed on oss2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 20: monitor MGS_1_monitor_120000 on oss2 Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_start_0 (23) confirmed on mds2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 24: monitor lustre-OST0001_4_monitor_120000 on mds2 Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_start_0 (25) confirmed on mds2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 26: monitor lustre-OST0000_3_monitor_120000 on mds2 Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_monitor_120000 (22) confirmed on oss1 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_monitor_120000 (20) confirmed on oss2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_monitor_120000 (24) confirmed on mds2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_monitor_120000 (26) confirmed on mds2 (rc=0) Jan 4 10:23:38 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 10:23:38 mds1 crmd: [1305]: notice: run_graph: Transition 99 (Complete=13, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-37.bz2): Complete Jan 4 10:23:38 mds1 crmd: [1305]: info: te_graph_trigger: Transition 99 is now complete Jan 4 10:23:38 mds1 crmd: [1305]: info: notify_crmd: Transition 99 status: done - Jan 4 10:23:38 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 10:23:38 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 10:31:01 mds1 cib: [1301]: info: cib_stats: Processed 45 operations (3111.00us average, 0% utilization) in the last 10min Jan 4 10:38:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:38:38 mds1 crmd: [1305]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped! Jan 4 10:38:38 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 10:38:38 mds1 crmd: [1305]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED Jan 4 10:38:38 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 10:38:38 mds1 crmd: [1305]: info: do_pe_invoke: Query 281: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:38:38 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=281, ref=pe_calc-dc-1325691518-247, seq=5028, quorate=1 Jan 4 10:38:38 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:38:38 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:38:38 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:38:38 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:38:38 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 10:38:38 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 10:38:38 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:38:38 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:38:38 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds2: unknown exec error (-2) Jan 4 10:38:38 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:38:38 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds2: unknown exec error (-2) Jan 4 10:38:38 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 10:38:38 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:38:38 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:38:38 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:38:38 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:38:38 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss2: unknown exec error (-2) Jan 4 10:38:38 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss2: unknown exec error (-2) Jan 4 10:38:38 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:38:38 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:38:38 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:38:38 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:38:38 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:38:38 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started oss2 Jan 4 10:38:38 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:38:38 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:38:38 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:38:38 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss1 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:38:38 mds1 pengine: [1304]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss1 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:38:38 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss2 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss2 after 1000000 failures (max=1000000) Jan 4 10:38:38 mds1 pengine: [1304]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss2 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss2 after 1000000 failures (max=1000000) Jan 4 10:38:38 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds2 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds2 after 1000000 failures (max=1000000) Jan 4 10:38:38 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds2 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds2 after 1000000 failures (max=1000000) Jan 4 10:38:38 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:38:38 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:38:38 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:38:38 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose mds1. Jan 4 10:38:38 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose mds1. Jan 4 10:38:38 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:38:38 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 10:38:38 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started mds1) Jan 4 10:38:38 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started mds1) Jan 4 10:38:38 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:38:38 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 10:38:38 mds1 pengine: [1304]: notice: LogActions: Leave resource MGS_1#011(Started oss2) Jan 4 10:38:38 mds1 pengine: [1304]: notice: LogActions: Leave resource lustre-MDT0000_2#011(Started oss1) Jan 4 10:38:38 mds1 pengine: [1304]: notice: LogActions: Leave resource lustre-OST0001_4#011(Started mds2) Jan 4 10:38:38 mds1 pengine: [1304]: notice: LogActions: Leave resource lustre-OST0000_3#011(Started mds2) Jan 4 10:38:38 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 10:38:38 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 100: 0 actions in 0 synapses Jan 4 10:38:38 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 100 (ref=pe_calc-dc-1325691518-247) derived from /var/lib/pengine/pe-input-38.bz2 Jan 4 10:38:38 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 10:38:38 mds1 crmd: [1305]: notice: run_graph: Transition 100 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pengine/pe-input-38.bz2): Complete Jan 4 10:38:38 mds1 crmd: [1305]: info: te_graph_trigger: Transition 100 is now complete Jan 4 10:38:38 mds1 crmd: [1305]: info: notify_crmd: Transition 100 status: done - Jan 4 10:38:38 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 10:38:38 mds1 crmd: [1305]: info: do_state_transition: Starting PEngine Recheck Timer Jan 4 10:38:38 mds1 pengine: [1304]: info: process_pe_message: Transition 100: PEngine Input stored in: /var/lib/pengine/pe-input-38.bz2 Jan 4 10:38:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:38:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:39:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:39:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:39:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:39:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:40:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:40:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:40:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:40:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:41:01 mds1 cib: [1301]: info: cib_stats: Processed 4 operations (15000.00us average, 0% utilization) in the last 10min Jan 4 10:41:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:41:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:41:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:42:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:42:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:42:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:42:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:42:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:42:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:43:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:43:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:43:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:44:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:44:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:44:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:44:42 mds1 corosync[1294]: [SERV ] Unloading all Corosync service engines. Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_shutdown: Shuting down Pacemaker Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: stop_child: Sent -15 to crmd: [1305] Jan 4 10:44:42 mds1 crmd: [1305]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 10:44:42 mds1 crmd: [1305]: info: crm_shutdown: Requesting shutdown Jan 4 10:44:42 mds1 crmd: [1305]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ] Jan 4 10:44:42 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 10:44:42 mds1 crmd: [1305]: info: do_shutdown_req: Sending shutdown request to DC: mds1 Jan 4 10:44:42 mds1 crmd: [1305]: info: handle_shutdown_request: Creating shutdown request for mds1 (state=S_POLICY_ENGINE) Jan 4 10:44:42 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (1325691881) Jan 4 10:44:42 mds1 attrd: [1303]: info: attrd_perform_update: Sent update 124: shutdown=1325691881 Jan 4 10:44:42 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=mds1, magic=NA, cib=0.246.43) : Transient attribute: update Jan 4 10:44:42 mds1 crmd: [1305]: info: do_pe_invoke: Query 283: Requesting the current CIB: S_POLICY_ENGINE Jan 4 10:44:42 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=283, ref=pe_calc-dc-1325691882-249, seq=5028, quorate=1 Jan 4 10:44:42 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 10:44:42 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 10:44:42 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 10:44:42 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 10:44:42 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 10:44:42 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 10:44:42 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 10:44:42 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 10:44:42 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds2: unknown exec error (-2) Jan 4 10:44:42 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 10:44:42 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds2: unknown exec error (-2) Jan 4 10:44:42 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is shutting down Jan 4 10:44:42 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 10:44:42 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 10:44:42 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 10:44:42 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 10:44:42 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss2: unknown exec error (-2) Jan 4 10:44:42 mds1 pengine: [1304]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss2: unknown exec error (-2) Jan 4 10:44:42 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 10:44:42 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:44:42 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:44:42 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started oss1 Jan 4 10:44:42 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 10:44:42 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started oss2 Jan 4 10:44:42 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started oss1 Jan 4 10:44:42 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:44:42 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started mds2 Jan 4 10:44:42 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss1 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:44:42 mds1 pengine: [1304]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss1 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss1 after 1000000 failures (max=1000000) Jan 4 10:44:42 mds1 pengine: [1304]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss2 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss2 after 1000000 failures (max=1000000) Jan 4 10:44:42 mds1 pengine: [1304]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss2 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss2 after 1000000 failures (max=1000000) Jan 4 10:44:42 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds2 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds2 after 1000000 failures (max=1000000) Jan 4 10:44:42 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds2 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds2 after 1000000 failures (max=1000000) Jan 4 10:44:42 mds1 pengine: [1304]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:44:42 mds1 pengine: [1304]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 10:44:42 mds1 pengine: [1304]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 10:44:42 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 10:44:42 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 10:44:42 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose oss1. Jan 4 10:44:42 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose oss2. Jan 4 10:44:42 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on oss1 Jan 4 10:44:42 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on oss2 Jan 4 10:44:42 mds1 pengine: [1304]: info: stage6: Scheduling Node mds1 for shutdown Jan 4 10:44:42 mds1 pengine: [1304]: notice: LogActions: Move resource st-oss2#011(Started mds1 -> oss1) Jan 4 10:44:42 mds1 pengine: [1304]: notice: LogActions: Move resource st-oss1#011(Started mds1 -> oss2) Jan 4 10:44:42 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started oss1) Jan 4 10:44:42 mds1 pengine: [1304]: notice: LogActions: Move resource st-mds2#011(Started mds1 -> oss2) Jan 4 10:44:42 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started oss2 -> oss1) Jan 4 10:44:42 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started oss1 -> oss2) Jan 4 10:44:42 mds1 pengine: [1304]: notice: LogActions: Leave resource lustre-OST0001_4#011(Started mds2) Jan 4 10:44:42 mds1 pengine: [1304]: notice: LogActions: Leave resource lustre-OST0000_3#011(Started mds2) Jan 4 10:44:42 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 10:44:42 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 101: 14 actions in 14 synapses Jan 4 10:44:42 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 101 (ref=pe_calc-dc-1325691882-249) derived from /var/lib/pengine/pe-input-39.bz2 Jan 4 10:44:42 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 11: stop st-oss2_stop_0 on mds1 (local) Jan 4 10:44:42 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=11:101:0:d3f0e147-de8d-45fc-bf26-c4214c060be6 op=st-oss2_stop_0 ) Jan 4 10:44:42 mds1 lrmd: [1302]: info: rsc:st-oss2:21: stop Jan 4 10:44:42 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 13: stop st-oss1_stop_0 on mds1 (local) Jan 4 10:44:42 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=13:101:0:d3f0e147-de8d-45fc-bf26-c4214c060be6 op=st-oss1_stop_0 ) Jan 4 10:44:42 mds1 lrmd: [1302]: info: rsc:st-oss1:22: stop Jan 4 10:44:42 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 17: stop st-mds2_stop_0 on mds1 (local) Jan 4 10:44:42 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=17:101:0:d3f0e147-de8d-45fc-bf26-c4214c060be6 op=st-mds2_stop_0 ) Jan 4 10:44:42 mds1 lrmd: [1302]: info: rsc:st-mds2:23: stop Jan 4 10:44:42 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 19: stop MGS_1_stop_0 on oss2 Jan 4 10:44:42 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 22: stop lustre-MDT0000_2_stop_0 on oss1 Jan 4 10:44:42 mds1 crmd: [1305]: info: handle_shutdown_request: Creating shutdown request for oss2 (state=S_TRANSITION_ENGINE) Jan 4 10:44:42 mds1 stonith-ng: [1300]: info: stonith_device_remove: Removed 'st-oss2' from the device list (2 active devices) Jan 4 10:44:42 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_device_remove from lrmd: rc=0 Jan 4 10:44:42 mds1 stonith-ng: [1300]: info: stonith_device_remove: Removed 'st-oss1' from the device list (1 active devices) Jan 4 10:44:42 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_device_remove from lrmd: rc=0 Jan 4 10:44:42 mds1 stonith-ng: [1300]: info: stonith_device_remove: Removed 'st-mds2' from the device list (0 active devices) Jan 4 10:44:42 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_device_remove from lrmd: rc=0 Jan 4 10:44:42 mds1 crmd: [1305]: info: process_lrm_event: LRM operation st-oss2_stop_0 (call=21, rc=0, cib-update=285, confirmed=true) ok Jan 4 10:44:42 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_stop_0 (11) confirmed on mds1 (rc=0) Jan 4 10:44:42 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 12: start st-oss2_start_0 on oss1 Jan 4 10:44:42 mds1 crmd: [1305]: info: process_lrm_event: LRM operation st-oss1_stop_0 (call=22, rc=0, cib-update=286, confirmed=true) ok Jan 4 10:44:42 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_stop_0 (13) confirmed on mds1 (rc=0) Jan 4 10:44:42 mds1 crmd: [1305]: info: handle_shutdown_request: Creating shutdown request for mds2 (state=S_TRANSITION_ENGINE) Jan 4 10:44:42 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 14: start st-oss1_start_0 on oss2 Jan 4 10:44:42 mds1 crmd: [1305]: info: process_lrm_event: LRM operation st-mds2_stop_0 (call=23, rc=0, cib-update=288, confirmed=true) ok Jan 4 10:44:42 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_stop_0 (17) confirmed on mds1 (rc=0) Jan 4 10:44:42 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 18: start st-mds2_start_0 on oss2 Jan 4 10:44:42 mds1 crmd: [1305]: info: te_crm_command: Executing crm-event (31): do_shutdown on mds1 Jan 4 10:44:42 mds1 crmd: [1305]: info: te_crm_command: crm-event (31) is a local shutdown Jan 4 10:44:42 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss2, magic=NA, cib=0.246.49) : Transient attribute: update Jan 4 10:44:42 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Jan 4 10:44:42 mds1 crmd: [1305]: info: update_abort_priority: 'local shutdown' abort superceeded Jan 4 10:44:42 mds1 pengine: [1304]: info: process_pe_message: Transition 101: PEngine Input stored in: /var/lib/pengine/pe-input-39.bz2 Jan 4 10:44:42 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.246.50) : Transient attribute: update Jan 4 10:44:42 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_start_0 (12) confirmed on oss1 (rc=0) Jan 4 10:44:42 mds1 crmd: [1305]: info: handle_shutdown_request: Creating shutdown request for oss1 (state=S_TRANSITION_ENGINE) Jan 4 10:44:42 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_start_0 (18) confirmed on oss2 (rc=0) Jan 4 10:44:42 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_start_0 (14) confirmed on oss2 (rc=0) Jan 4 10:44:42 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.246.55) : Transient attribute: update Jan 4 10:44:42 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_stop_0 (22) confirmed on oss1 (rc=0) Jan 4 10:44:42 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_stop_0 (19) confirmed on oss2 (rc=0) Jan 4 10:44:42 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 10:44:42 mds1 crmd: [1305]: notice: run_graph: Transition 101 (Complete=9, Pending=0, Fired=0, Skipped=5, Incomplete=0, Source=/var/lib/pengine/pe-input-39.bz2): Stopped Jan 4 10:44:42 mds1 crmd: [1305]: info: te_graph_trigger: Transition 101 is now complete Jan 4 10:44:42 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_STOPPING [ input=I_STOP cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 10:44:42 mds1 crmd: [1305]: info: do_dc_release: DC role released Jan 4 10:44:42 mds1 crmd: [1305]: info: pe_connection_destroy: Connection to the Policy Engine released Jan 4 10:44:42 mds1 crmd: [1305]: info: do_te_control: Transitioner is now inactive Jan 4 10:44:42 mds1 crmd: [1305]: info: do_shutdown: Disconnecting STONITH... Jan 4 10:44:42 mds1 crmd: [1305]: info: tengine_stonith_connection_destroy: Fencing daemon disconnected Jan 4 10:44:42 mds1 crmd: [1305]: info: do_lrm_control: Disconnected from the LRM Jan 4 10:44:42 mds1 crmd: [1305]: info: do_ha_control: Disconnected from OpenAIS Jan 4 10:44:42 mds1 crmd: [1305]: info: do_cib_control: Disconnecting CIB Jan 4 10:44:42 mds1 crmd: [1305]: info: crmd_cib_connection_destroy: Connection to the CIB terminated... Jan 4 10:44:42 mds1 cib: [1301]: info: cib_process_readwrite: We are now in R/O mode Jan 4 10:44:42 mds1 crmd: [1305]: info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd Jan 4 10:44:42 mds1 cib: [1301]: WARN: send_ipc_message: IPC Channel to 1305 is not connected Jan 4 10:44:42 mds1 crmd: [1305]: info: free_mem: Dropping I_RELEASE_SUCCESS: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_dc_release ] Jan 4 10:44:42 mds1 cib: [1301]: WARN: send_via_callback_channel: Delivery of reply to client 1305/b2461ae2-7bcc-468c-b5b9-703df407ef74 failed Jan 4 10:44:42 mds1 crmd: [1305]: info: free_mem: Dropping I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ] Jan 4 10:44:42 mds1 cib: [1301]: WARN: do_local_notify: A-Sync reply to crmd failed: reply failed Jan 4 10:44:42 mds1 crmd: [1305]: info: do_exit: [crmd] stopped (0) Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] info: pcmk_ipc_exit: Client crmd (conn=0xee0ee0, async-conn=0xee0ee0) left Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_shutdown: crmd confirmed stopped Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: stop_child: Sent -15 to pengine: [1304] Jan 4 10:44:42 mds1 pengine: [1304]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_shutdown: pengine confirmed stopped Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: stop_child: Sent -15 to attrd: [1303] Jan 4 10:44:42 mds1 attrd: [1303]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 10:44:42 mds1 attrd: [1303]: info: attrd_shutdown: Exiting Jan 4 10:44:42 mds1 attrd: [1303]: info: main: Exiting... Jan 4 10:44:42 mds1 attrd: [1303]: info: attrd_cib_connection_destroy: Connection to the CIB terminated... Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_shutdown: attrd confirmed stopped Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: stop_child: Sent -15 to lrmd: [1302] Jan 4 10:44:42 mds1 lrmd: [1302]: info: lrmd is shutting down Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_shutdown: lrmd confirmed stopped Jan 4 10:44:42 mds1 cib: [1301]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: stop_child: Sent -15 to cib: [1301] Jan 4 10:44:42 mds1 cib: [1301]: info: cib_shutdown: Disconnected 0 clients Jan 4 10:44:42 mds1 cib: [1301]: info: cib_process_disconnect: All clients disconnected... Jan 4 10:44:42 mds1 cib: [1301]: info: initiate_exit: Sending disconnect notification to 4 peers... Jan 4 10:44:42 mds1 cib: [1301]: info: cib_process_shutdown_req: Shutdown ACK from oss2 Jan 4 10:44:42 mds1 cib: [1301]: info: cib_ha_connection_destroy: Heartbeat disconnection complete... exiting Jan 4 10:44:42 mds1 cib: [1301]: info: cib_ha_connection_destroy: Exiting... Jan 4 10:44:42 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_shutdown_req for section 'all' (origin=oss2/oss2/(null), version=0.0.0): ok (rc=0) Jan 4 10:44:42 mds1 cib: [1301]: info: main: Done Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_shutdown: cib confirmed stopped Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: stop_child: Sent -15 to stonith-ng: [1300] Jan 4 10:44:42 mds1 stonith-ng: [1300]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 10:44:42 mds1 stonith-ng: [1300]: info: stonith_shutdown: Terminating with 0 clients Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_shutdown: stonith-ng confirmed stopped Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] info: update_member: Node mds1 now has process list: 00000000000000000000000000000002 (2) Jan 4 10:44:42 mds1 corosync[1294]: [pcmk ] notice: pcmk_shutdown: Shutdown complete Jan 4 10:44:42 mds1 corosync[1294]: [SERV ] Service engine unloaded: Pacemaker Cluster Manager 1.1.2 Jan 4 10:44:42 mds1 corosync[1294]: [SERV ] Service engine unloaded: corosync extended virtual synchrony service Jan 4 10:44:42 mds1 corosync[1294]: [SERV ] Service engine unloaded: corosync configuration service Jan 4 10:44:42 mds1 corosync[1294]: [SERV ] Service engine unloaded: corosync cluster closed process group service v1.01 Jan 4 10:44:42 mds1 corosync[1294]: [SERV ] Service engine unloaded: corosync cluster config database access v1.01 Jan 4 10:44:42 mds1 corosync[1294]: [SERV ] Service engine unloaded: corosync profile loading service Jan 4 10:44:42 mds1 corosync[1294]: [SERV ] Service engine unloaded: corosync cluster quorum service v0.1 Jan 4 10:44:42 mds1 corosync[1294]: [MAIN ] Corosync Cluster Engine exiting with status 0 at main.c:170. Jan 4 10:44:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:44:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:45:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:45:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:45:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:45:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:45:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:45:53 mds1 kernel: init dynlocks cache Jan 4 10:45:53 mds1 kernel: LDISKFS-fs warning (device sdb): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs. Jan 4 10:45:53 mds1 kernel: LDISKFS-fs (sdb): mounted filesystem with ordered data mode Jan 4 10:45:53 mds1 kernel: LDISKFS-fs warning (device sdb): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs. Jan 4 10:45:53 mds1 kernel: LDISKFS-fs (sdb): mounted filesystem with ordered data mode Jan 4 10:45:53 mds1 kernel: Lustre: Added LNI 192.168.122.89@tcp [8/256/0/180] Jan 4 10:45:53 mds1 kernel: Lustre: Accept secure, port 988 Jan 4 10:45:53 mds1 kernel: Lustre: MGS MGS started Jan 4 10:45:53 mds1 kernel: Lustre: 4258:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGC192.168.122.89@tcp->MGC192.168.122.89@tcp_0 netid 90000: select flavor null Jan 4 10:45:53 mds1 kernel: Lustre: 4296:0:(ldlm_lib.c:877:target_handle_connect()) MGS: connection from 439e3ac6-a669-fe23-ec81-7e440b074dfe@0@lo t0 exp (null) cur 1325691953 last 0 Jan 4 10:45:53 mds1 kernel: Lustre: 4296:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x9000000000000_UUID netid 90000: select flavor null Jan 4 10:45:53 mds1 kernel: Lustre: MGC192.168.122.89@tcp: Reactivating import Jan 4 10:45:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:46:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:46:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:46:28 mds1 kernel: Lustre: 4296:0:(ldlm_lib.c:877:target_handle_connect()) MGS: connection from a4443e1b-1934-f3d3-5658-1071ac47f697@192.168.122.155@tcp t0 exp (null) cur 1325691988 last 0 Jan 4 10:46:28 mds1 kernel: Lustre: 4296:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x20000c0a87a9b_UUID netid 20000: select flavor null Jan 4 10:46:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:46:44 mds1 kernel: Lustre: 4297:0:(ldlm_lib.c:877:target_handle_connect()) MGS: connection from bed2a361-1cce-faaf-0722-7b3607cec24b@192.168.122.147@tcp t0 exp (null) cur 1325692004 last 0 Jan 4 10:46:44 mds1 kernel: Lustre: 4297:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x20000c0a87a93_UUID netid 20000: select flavor null Jan 4 10:46:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:46:59 mds1 kernel: Lustre: 4297:0:(ldlm_lib.c:877:target_handle_connect()) MGS: connection from 955bbd8b-395a-94e4-f103-87786de5eec4@192.168.122.160@tcp t0 exp (null) cur 1325692019 last 0 Jan 4 10:46:59 mds1 kernel: Lustre: 4297:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x20000c0a87aa0_UUID netid 20000: select flavor null Jan 4 10:47:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:47:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:47:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:47:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:47:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:48:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:48:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:48:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:48:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:49:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:49:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:49:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:50:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:50:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:50:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:50:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:51:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:51:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:51:29 mds1 kernel: Lustre: 4296:0:(ldlm_lib.c:877:target_handle_connect()) MGS: connection from eeb1b360-5b93-74de-6b1b-d67428a6b341@192.168.122.218@tcp t0 exp (null) cur 1325692289 last 0 Jan 4 10:51:29 mds1 kernel: Lustre: 4296:0:(sec.c:1474:sptlrpc_import_sec_adapt()) import MGS->NET_0x20000c0a87ada_UUID netid 20000: select flavor null Jan 4 10:51:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:51:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:52:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:52:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:52:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:52:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:52:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:53:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:53:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:53:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:53:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:54:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:54:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:54:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:54:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:54:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:55:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:55:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:55:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:56:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:56:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:56:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:56:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:57:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:57:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:57:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:57:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:58:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:58:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:58:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:58:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:58:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:59:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:59:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:59:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 10:59:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:00:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:00:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:00:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:00:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:01:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:01:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:01:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:01:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:02:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:02:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:02:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:02:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:02:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:03:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:03:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:03:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:03:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:03:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:04:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:04:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:04:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:04:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:04:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:05:12 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 11:05:12 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 11:05:12 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1573 seconds. Jan 4 11:31:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:31:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:31:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:32:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:32:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:32:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:32:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:32:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:33:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:33:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:33:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:33:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:33:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:34:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:34:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:34:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:34:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:35:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:35:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:35:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:35:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:36:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:36:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:36:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:36:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:36:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:37:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:37:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:37:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:37:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:38:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:38:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:38:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:38:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:38:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:39:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:39:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:39:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:39:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:40:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:40:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:40:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:40:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:41:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:41:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:41:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:41:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:41:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:42:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:42:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:42:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:42:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:42:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:43:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:43:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:43:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:44:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:44:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:44:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:44:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:45:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:45:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:45:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:45:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:46:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:46:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:46:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:46:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:47:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:47:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:47:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:47:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:48:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:48:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:48:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:48:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:48:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:49:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:49:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:49:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:49:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:49:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:49:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:50:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:50:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:50:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:51:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:51:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:51:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:51:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:52:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:52:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:52:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:52:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:53:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:53:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:53:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:53:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:53:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:53:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:54:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:54:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:54:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 11:54:35 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 11:54:35 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 11:54:35 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1609 seconds. Jan 4 12:21:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:21:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:21:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:21:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:22:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:22:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:22:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:22:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:23:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:23:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:23:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:23:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:23:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:24:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:24:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:24:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:25:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:25:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:25:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:25:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:26:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:26:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:26:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:26:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:27:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:27:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:27:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:27:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:28:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:28:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:28:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:28:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:28:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:28:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:29:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:29:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:29:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:30:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:30:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:30:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:30:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:30:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:31:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:31:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:31:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:31:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:32:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:32:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:32:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:32:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:32:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:33:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:33:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:33:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:33:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:33:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:34:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:34:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:34:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:34:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:34:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:35:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:35:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:35:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:35:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:36:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:36:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:36:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:36:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:37:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:37:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:37:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:37:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:37:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:38:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:38:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:38:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:39:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:39:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:39:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:39:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:39:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:40:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:40:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:40:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:40:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:40:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:41:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:41:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:41:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:41:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:42:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:42:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:42:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:42:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:42:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:43:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:43:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:43:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:43:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:43:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:44:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:44:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:44:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 12:44:54 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 12:44:54 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 12:44:54 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1318 seconds. Jan 4 13:06:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:06:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:07:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:07:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:07:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:07:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:07:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:08:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:08:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:08:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:08:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:08:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:08:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:09:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:09:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:09:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:09:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:09:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:10:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:10:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:10:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:10:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:11:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:11:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:11:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:11:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:11:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:12:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:12:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:12:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:12:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:12:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:12:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:13:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:13:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:13:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:13:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:14:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:14:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:14:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:14:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:15:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:15:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:15:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:16:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:16:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:16:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:16:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:17:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:17:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:17:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:17:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:17:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:18:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:18:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:18:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:18:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:19:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:19:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:19:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:19:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:20:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:20:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:20:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:20:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:21:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:21:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:21:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:22:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:22:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:22:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:22:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:22:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:22:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:23:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:23:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:23:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:23:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:23:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:24:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:24:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:24:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:24:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:25:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:25:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:25:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:25:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:26:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:26:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:26:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:26:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:26:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:27:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:27:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:27:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:28:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:28:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:28:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:28:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:28:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:29:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:29:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:29:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:29:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:30:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:30:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:30:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:30:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:31:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:31:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:31:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:32:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:32:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:32:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:32:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:32:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:32:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:33:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:33:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:33:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:34:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:34:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:34:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:34:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:34:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:35:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:35:26 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 13:35:26 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 13:35:26 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1286 seconds. Jan 4 13:56:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:56:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:57:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:57:28 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:57:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:58:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:58:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:58:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:58:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:59:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:59:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:59:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:59:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 13:59:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:00:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:00:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:00:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:00:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:00:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:01:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:01:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:01:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:01:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:02:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:02:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:02:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:02:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:02:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:03:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:03:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:03:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:03:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:04:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:04:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:04:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:04:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:04:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:05:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:05:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:05:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:05:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:05:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:06:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:06:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:06:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:06:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:06:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:07:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:07:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:07:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:07:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:08:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:08:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:08:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:08:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:08:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:09:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:09:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:09:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:09:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:09:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:10:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:10:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:10:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:10:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:10:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:10:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:11:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:11:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:11:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:11:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:11:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:12:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:12:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:12:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:12:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:13:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:13:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:13:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:13:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:14:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:14:19 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:14:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:14:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:15:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:15:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:15:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:15:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:16:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:16:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:16:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:17:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:17:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:17:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:17:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:17:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:18:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:18:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:18:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:18:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:19:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:19:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:19:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:19:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:19:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:20:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:20:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:20:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:20:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:20:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:21:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:21:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:21:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:21:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:21:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:22:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:22:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:22:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:22:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:23:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:23:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:23:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:23:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:24:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:24:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:24:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:24:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:24:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:25:13 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:25:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:25:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:25:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:26:06 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 14:26:06 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 14:26:06 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1258 seconds. Jan 4 14:47:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:47:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:47:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:47:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:47:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:47:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:48:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:48:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:48:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:48:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:49:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:49:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:49:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:49:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:49:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:50:03 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:50:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:50:37 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:50:53 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:51:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:51:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:51:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:51:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:52:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:52:11 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:52:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:52:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:53:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:53:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:53:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:53:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:53:54 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:54:02 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:54:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:54:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:54:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:55:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:55:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:55:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:55:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:56:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:56:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:56:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:56:50 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:57:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:57:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:57:27 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:57:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:57:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:57:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:58:06 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:58:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:58:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:58:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:59:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:59:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:59:43 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 14:59:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:00:14 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:00:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:00:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:00:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:00:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:01:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:01:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:01:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:01:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:02:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:02:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:02:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:02:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:02:45 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:02:57 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:03:08 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:03:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:03:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:03:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:03:51 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:04:12 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:04:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:04:39 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:04:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:04:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:05:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:05:24 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:05:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:05:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:06:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:06:18 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:06:33 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:06:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:06:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:07:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:07:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:07:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:08:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:08:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:08:40 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:08:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:08:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:09:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:09:26 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:09:36 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:09:46 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:10:00 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:10:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:10:32 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:10:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:11:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:11:21 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:11:29 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:11:38 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:11:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:12:05 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:12:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:12:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:12:48 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:12:55 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:13:04 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:13:15 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:13:25 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:13:41 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:13:58 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:14:10 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:14:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:14:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:14:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:15:01 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:15:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:15:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:15:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:16:06 mds1 dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67 Jan 4 15:16:06 mds1 dhclient: DHCPACK from 192.168.122.1 Jan 4 15:16:06 mds1 dhclient: bound to 192.168.122.89 -- renewal in 1331 seconds. Jan 4 15:38:17 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:38:20 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:38:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:38:31 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:38:52 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:39:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:39:16 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:39:30 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:39:42 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:39:49 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:39:56 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:40:07 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:40:23 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:40:35 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:40:44 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:40:59 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:41:09 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:41:22 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:41:34 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:41:47 mds1 dhclient: DHCPREQUEST on eth0 to 192.168.122.1 port 67 Jan 4 15:41:59 mds1 init: tty (/dev/tty1) main process (1578) killed by TERM signal Jan 4 15:41:59 mds1 init: tty (/dev/tty2) main process (1580) killed by TERM signal Jan 4 15:41:59 mds1 init: tty (/dev/tty3) main process (1582) killed by TERM signal Jan 4 15:41:59 mds1 init: tty (/dev/tty4) main process (1584) killed by TERM signal Jan 4 15:41:59 mds1 init: tty (/dev/tty5) main process (1586) killed by TERM signal Jan 4 15:41:59 mds1 init: tty (/dev/tty6) main process (1588) killed by TERM signal Jan 4 15:41:59 mds1 avahi-dnsconfd[1557]: Got SIGTERM, quitting. Jan 4 15:42:00 mds1 acpid: exiting Jan 4 15:42:00 mds1 init: Disconnected from system bus Jan 4 15:42:00 mds1 console-kit-daemon[4484]: WARNING: no sender#012 Jan 4 15:42:00 mds1 rpcbind: rpcbind terminating on signal. Restart with "rpcbind -w" Jan 4 15:42:00 mds1 auditd[1606]: The audit daemon is exiting. Jan 4 15:42:00 mds1 kernel: type=1305 audit(1325709720.740:18195): audit_pid=0 old=1606 auid=4294967295 ses=4294967295 res=1 Jan 4 15:42:00 mds1 NET[4770]: /sbin/dhclient-script : updated /etc/resolv.conf Jan 4 15:42:01 mds1 kernel: IPv6 over IPv4 tunneling driver Jan 4 15:42:01 mds1 kernel: sit0: Disabled Privacy Extensions Jan 4 15:42:01 mds1 kernel: Kernel logging (proc) stopped. Jan 4 15:42:01 mds1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1168" x-info="http://www.rsyslog.com"] exiting on signal 15. Jan 4 16:46:41 mds1 kernel: imklog 4.6.2, log source = /proc/kmsg started. Jan 4 16:46:41 mds1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1164" x-info="http://www.rsyslog.com"] (re)start Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys cpuset Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys cpu Jan 4 16:46:41 mds1 kernel: Linux version 2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 (jenkins@rhel6-64-build01.lab.whamcloud.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Sat Aug 20 09:56:49 PDT 2011 Jan 4 16:46:41 mds1 kernel: Command line: ro root=/dev/mapper/LustreVG-root rd_LVM_LV=LustreVG/root rd_LVM_LV=LustreVG/swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet Jan 4 16:46:41 mds1 kernel: KERNEL supported cpus: Jan 4 16:46:41 mds1 kernel: Intel GenuineIntel Jan 4 16:46:41 mds1 kernel: AMD AuthenticAMD Jan 4 16:46:41 mds1 kernel: Centaur CentaurHauls Jan 4 16:46:41 mds1 kernel: BIOS-provided physical RAM map: Jan 4 16:46:41 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) Jan 4 16:46:41 mds1 kernel: BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) Jan 4 16:46:41 mds1 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) Jan 4 16:46:41 mds1 kernel: BIOS-e820: 0000000000100000 - 000000003fffd000 (usable) Jan 4 16:46:41 mds1 kernel: BIOS-e820: 000000003fffd000 - 0000000040000000 (reserved) Jan 4 16:46:41 mds1 kernel: BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved) Jan 4 16:46:41 mds1 kernel: BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved) Jan 4 16:46:41 mds1 kernel: DMI 2.4 present. Jan 4 16:46:41 mds1 kernel: SMBIOS version 2.4 @ 0xFD760 Jan 4 16:46:41 mds1 kernel: last_pfn = 0x3fffd max_arch_pfn = 0x400000000 Jan 4 16:46:41 mds1 kernel: PAT not supported by CPU. Jan 4 16:46:41 mds1 kernel: init_memory_mapping: 0000000000000000-000000003fffd000 Jan 4 16:46:41 mds1 kernel: RAMDISK: 34a6c000 - 37fef97b Jan 4 16:46:41 mds1 kernel: ACPI: RSDP 00000000000fd730 00014 (v00 BOCHS ) Jan 4 16:46:41 mds1 kernel: ACPI: RSDT 000000003fffdc40 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) Jan 4 16:46:41 mds1 kernel: ACPI: FACP 000000003ffffe70 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) Jan 4 16:46:41 mds1 kernel: ACPI: DSDT 000000003fffde40 01FB7 (v01 BXPC BXDSDT 00000001 INTL 20090123) Jan 4 16:46:41 mds1 kernel: ACPI: FACS 000000003ffffe00 00040 Jan 4 16:46:41 mds1 kernel: ACPI: SSDT 000000003fffdda0 0009E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) Jan 4 16:46:41 mds1 kernel: ACPI: APIC 000000003fffdcc0 00072 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) Jan 4 16:46:41 mds1 kernel: ACPI: HPET 000000003fffdc80 00038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001) Jan 4 16:46:41 mds1 kernel: No NUMA configuration found Jan 4 16:46:41 mds1 kernel: Faking a node at 0000000000000000-000000003fffd000 Jan 4 16:46:41 mds1 kernel: Bootmem setup node 0 0000000000000000-000000003fffd000 Jan 4 16:46:41 mds1 kernel: NODE_DATA [0000000000009000 - 000000000003cfff] Jan 4 16:46:41 mds1 kernel: bootmap [000000000003d000 - 0000000000044fff] pages 8 Jan 4 16:46:41 mds1 kernel: (7 early reservations) ==> bootmem [0000000000 - 003fffd000] Jan 4 16:46:41 mds1 kernel: #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] Jan 4 16:46:41 mds1 kernel: #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] Jan 4 16:46:41 mds1 kernel: #2 [0001000000 - 0001f474e4] TEXT DATA BSS ==> [0001000000 - 0001f474e4] Jan 4 16:46:41 mds1 kernel: #3 [0034a6c000 - 0037fef97b] RAMDISK ==> [0034a6c000 - 0037fef97b] Jan 4 16:46:41 mds1 kernel: #4 [000009dc00 - 0000100000] BIOS reserved ==> [000009dc00 - 0000100000] Jan 4 16:46:41 mds1 kernel: #5 [0001f48000 - 0001f48079] BRK ==> [0001f48000 - 0001f48079] Jan 4 16:46:41 mds1 kernel: #6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000] Jan 4 16:46:41 mds1 kernel: found SMP MP-table at [ffff8800000fd780] fd780 Jan 4 16:46:41 mds1 kernel: kvm-clock: Using msrs 12 and 11 Jan 4 16:46:41 mds1 kernel: kvm-clock: cpu 0, msr 0:1bbbf01, boot clock Jan 4 16:46:41 mds1 kernel: Zone PFN ranges: Jan 4 16:46:41 mds1 kernel: DMA 0x00000001 -> 0x00001000 Jan 4 16:46:41 mds1 kernel: DMA32 0x00001000 -> 0x00100000 Jan 4 16:46:41 mds1 kernel: Normal 0x00100000 -> 0x00100000 Jan 4 16:46:41 mds1 kernel: Movable zone start PFN for each node Jan 4 16:46:41 mds1 kernel: early_node_map[2] active PFN ranges Jan 4 16:46:41 mds1 kernel: 0: 0x00000001 -> 0x0000009d Jan 4 16:46:41 mds1 kernel: 0: 0x00000100 -> 0x0003fffd Jan 4 16:46:41 mds1 kernel: ACPI: PM-Timer IO Port: 0xb008 Jan 4 16:46:41 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Jan 4 16:46:41 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) Jan 4 16:46:41 mds1 kernel: IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23 Jan 4 16:46:41 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Jan 4 16:46:41 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) Jan 4 16:46:41 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Jan 4 16:46:41 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) Jan 4 16:46:41 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) Jan 4 16:46:41 mds1 kernel: Using ACPI (MADT) for SMP configuration information Jan 4 16:46:41 mds1 kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000 Jan 4 16:46:41 mds1 kernel: SMP: Allowing 1 CPUs, 0 hotplug CPUs Jan 4 16:46:41 mds1 kernel: PM: Registered nosave memory: 000000000009d000 - 000000000009e000 Jan 4 16:46:41 mds1 kernel: PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 Jan 4 16:46:41 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 Jan 4 16:46:41 mds1 kernel: PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 Jan 4 16:46:41 mds1 kernel: Allocating PCI resources starting at 40000000 (gap: 40000000:beffc000) Jan 4 16:46:41 mds1 kernel: Booting paravirtualized kernel on KVM Jan 4 16:46:41 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 Jan 4 16:46:41 mds1 kernel: PERCPU: Embedded 30 pages/cpu @ffff880002000000 s92504 r8192 d22184 u2097152 Jan 4 16:46:41 mds1 kernel: pcpu-alloc: s92504 r8192 d22184 u2097152 alloc=1*2097152 Jan 4 16:46:41 mds1 kernel: pcpu-alloc: [0] 0 Jan 4 16:46:41 mds1 kernel: kvm-clock: cpu 0, msr 0:2015f01, primary cpu clock Jan 4 16:46:41 mds1 kernel: Built 1 zonelists in Node order, mobility grouping on. Total pages: 258355 Jan 4 16:46:41 mds1 kernel: Policy zone: DMA32 Jan 4 16:46:41 mds1 kernel: Kernel command line: ro root=/dev/mapper/LustreVG-root rd_LVM_LV=LustreVG/root rd_LVM_LV=LustreVG/swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet Jan 4 16:46:41 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes) Jan 4 16:46:41 mds1 kernel: Checking aperture... Jan 4 16:46:41 mds1 kernel: No AGP bridge found Jan 4 16:46:41 mds1 kernel: AMD-Vi disabled by default: pass amd_iommu=on to enable Jan 4 16:46:41 mds1 kernel: Memory: 962756k/1048564k available (5016k kernel code, 400k absent, 85408k reserved, 6904k data, 1232k init) Jan 4 16:46:41 mds1 kernel: Hierarchical RCU implementation. Jan 4 16:46:41 mds1 kernel: NR_IRQS:33024 nr_irqs:256 Jan 4 16:46:41 mds1 kernel: Console: colour VGA+ 80x25 Jan 4 16:46:41 mds1 kernel: console [tty0] enabled Jan 4 16:46:41 mds1 kernel: allocated 10485760 bytes of page_cgroup Jan 4 16:46:41 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Jan 4 16:46:41 mds1 kernel: HPET: 3 timers in total, 0 timers will be used for per-cpu timer Jan 4 16:46:41 mds1 kernel: Detected 1595.883 MHz processor. Jan 4 16:46:41 mds1 kernel: Calibrating delay loop (skipped) preset value.. 3191.76 BogoMIPS (lpj=1595883) Jan 4 16:46:41 mds1 kernel: pid_max: default: 32768 minimum: 301 Jan 4 16:46:41 mds1 kernel: Security Framework initialized Jan 4 16:46:41 mds1 kernel: SELinux: Initializing. Jan 4 16:46:41 mds1 kernel: Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Jan 4 16:46:41 mds1 kernel: Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) Jan 4 16:46:41 mds1 kernel: Mount-cache hash table entries: 256 Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys ns Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys cpuacct Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys memory Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys devices Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys freezer Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys net_cls Jan 4 16:46:41 mds1 kernel: Initializing cgroup subsys blkio Jan 4 16:46:41 mds1 kernel: mce: CPU supports 10 MCE banks Jan 4 16:46:41 mds1 kernel: alternatives: switching to unfair spinlock Jan 4 16:46:41 mds1 kernel: SMP alternatives: switching to UP code Jan 4 16:46:41 mds1 kernel: Freeing SMP alternatives: 33k freed Jan 4 16:46:41 mds1 kernel: ACPI: Core revision 20090903 Jan 4 16:46:41 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00 Jan 4 16:46:41 mds1 kernel: ftrace: allocating 20705 entries in 82 pages Jan 4 16:46:41 mds1 kernel: Setting APIC routing to flat Jan 4 16:46:41 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Jan 4 16:46:41 mds1 kernel: CPU0: Intel QEMU Virtual CPU version 0.14.0 stepping 03 Jan 4 16:46:41 mds1 kernel: APIC calibration not consistent with PM-Timer: 104ms instead of 100ms Jan 4 16:46:41 mds1 kernel: APIC delta adjusted to PM-Timer: 6249973 (6500022) Jan 4 16:46:41 mds1 kernel: Performance Events: unsupported p6 CPU model 2 no PMU driver, software events only. Jan 4 16:46:41 mds1 kernel: NMI watchdog disabled (cpu0): hardware events not enabled Jan 4 16:46:41 mds1 kernel: Brought up 1 CPUs Jan 4 16:46:41 mds1 kernel: Total of 1 processors activated (3191.76 BogoMIPS). Jan 4 16:46:41 mds1 kernel: devtmpfs: initialized Jan 4 16:46:41 mds1 kernel: regulator: core version 0.5 Jan 4 16:46:41 mds1 kernel: NET: Registered protocol family 16 Jan 4 16:46:41 mds1 kernel: ACPI: bus type pci registered Jan 4 16:46:41 mds1 kernel: PCI: Using configuration type 1 for base access Jan 4 16:46:41 mds1 kernel: bio: create slab at 0 Jan 4 16:46:41 mds1 kernel: ACPI: Interpreter enabled Jan 4 16:46:41 mds1 kernel: ACPI: (supports S0 S3 S4 S5) Jan 4 16:46:41 mds1 kernel: ACPI: Using IOAPIC for interrupt routing Jan 4 16:46:41 mds1 kernel: ACPI: No dock devices found. Jan 4 16:46:41 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) Jan 4 16:46:41 mds1 kernel: pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI Jan 4 16:46:41 mds1 kernel: pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0 Jan 4 16:46:41 mds1 kernel: HEST: Table is not found! Jan 4 16:46:41 mds1 kernel: vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none Jan 4 16:46:41 mds1 kernel: vgaarb: loaded Jan 4 16:46:41 mds1 kernel: SCSI subsystem initialized Jan 4 16:46:41 mds1 kernel: usbcore: registered new interface driver usbfs Jan 4 16:46:41 mds1 kernel: usbcore: registered new interface driver hub Jan 4 16:46:41 mds1 kernel: usbcore: registered new device driver usb Jan 4 16:46:41 mds1 kernel: PCI: Using ACPI for IRQ routing Jan 4 16:46:41 mds1 kernel: NetLabel: Initializing Jan 4 16:46:41 mds1 kernel: NetLabel: domain hash size = 128 Jan 4 16:46:41 mds1 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Jan 4 16:46:41 mds1 kernel: NetLabel: unlabeled traffic allowed by default Jan 4 16:46:41 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 Jan 4 16:46:41 mds1 kernel: hpet0: 3 comparators, 64-bit 100.000000 MHz counter Jan 4 16:46:41 mds1 kernel: Switching to clocksource kvm-clock Jan 4 16:46:41 mds1 kernel: pnp: PnP ACPI init Jan 4 16:46:41 mds1 kernel: ACPI: bus type pnp registered Jan 4 16:46:41 mds1 kernel: pnp: PnP ACPI: found 6 devices Jan 4 16:46:41 mds1 kernel: ACPI: ACPI bus type pnp unregistered Jan 4 16:46:41 mds1 kernel: NET: Registered protocol family 2 Jan 4 16:46:41 mds1 kernel: IP route cache hash table entries: 32768 (order: 6, 262144 bytes) Jan 4 16:46:41 mds1 kernel: TCP established hash table entries: 131072 (order: 9, 2097152 bytes) Jan 4 16:46:41 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Jan 4 16:46:41 mds1 kernel: TCP: Hash tables configured (established 131072 bind 65536) Jan 4 16:46:41 mds1 kernel: TCP reno registered Jan 4 16:46:41 mds1 kernel: NET: Registered protocol family 1 Jan 4 16:46:41 mds1 kernel: pci 0000:00:00.0: Limiting direct PCI/PCI transfers Jan 4 16:46:41 mds1 kernel: pci 0000:00:01.0: PIIX3: Enabling Passive Release Jan 4 16:46:41 mds1 kernel: pci 0000:00:01.0: Activating ISA DMA hang workarounds Jan 4 16:46:41 mds1 kernel: Trying to unpack rootfs image as initramfs... Jan 4 16:46:41 mds1 kernel: Freeing initrd memory: 54798k freed Jan 4 16:46:41 mds1 kernel: audit: initializing netlink socket (disabled) Jan 4 16:46:41 mds1 kernel: type=2000 audit(1325713556.118:1): initialized Jan 4 16:46:41 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages Jan 4 16:46:41 mds1 kernel: VFS: Disk quotas dquot_6.5.2 Jan 4 16:46:41 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Jan 4 16:46:41 mds1 kernel: msgmni has been set to 1987 Jan 4 16:46:41 mds1 kernel: alg: No test for stdrng (krng) Jan 4 16:46:41 mds1 kernel: ksign: Installing public key data Jan 4 16:46:41 mds1 kernel: Loading keyring Jan 4 16:46:41 mds1 kernel: - Added public key CBA5A3F3E494381 Jan 4 16:46:41 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key) Jan 4 16:46:41 mds1 kernel: - Added public key D4A26C9CCD09BEDA Jan 4 16:46:41 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program Jan 4 16:46:41 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) Jan 4 16:46:41 mds1 kernel: io scheduler noop registered Jan 4 16:46:41 mds1 kernel: io scheduler anticipatory registered Jan 4 16:46:41 mds1 kernel: io scheduler deadline registered (default) Jan 4 16:46:41 mds1 kernel: io scheduler cfq registered Jan 4 16:46:41 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Jan 4 16:46:41 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Jan 4 16:46:41 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [1] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [2] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [3] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [4] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [5] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [6] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [7] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [8] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [9] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [10] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [11] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [12] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [13] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [14] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [15] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [16] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [17] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [18] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [19] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [20] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [21] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [22] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [23] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [24] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [25] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [26] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [27] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [28] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [29] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [30] registered Jan 4 16:46:41 mds1 kernel: acpiphp: Slot [31] registered Jan 4 16:46:41 mds1 kernel: pci-stub: invalid id string "" Jan 4 16:46:41 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 Jan 4 16:46:41 mds1 kernel: ACPI: Power Button [PWRF] Jan 4 16:46:41 mds1 kernel: processor LNXCPU:00: registered as cooling_device0 Jan 4 16:46:41 mds1 kernel: ERST: Table is not found! Jan 4 16:46:41 mds1 kernel: Non-volatile memory driver v1.3 Jan 4 16:46:41 mds1 kernel: Linux agpgart interface v0.103 Jan 4 16:46:41 mds1 kernel: crash memory driver: version 1.1 Jan 4 16:46:41 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Jan 4 16:46:41 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jan 4 16:46:41 mds1 kernel: brd: module loaded Jan 4 16:46:41 mds1 kernel: loop: module loaded Jan 4 16:46:41 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1 Jan 4 16:46:41 mds1 kernel: Fixed MDIO Bus: probed Jan 4 16:46:41 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Jan 4 16:46:41 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Jan 4 16:46:41 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 Jan 4 16:46:41 mds1 kernel: uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 Jan 4 16:46:41 mds1 kernel: uhci_hcd 0000:00:01.2: UHCI Host Controller Jan 4 16:46:41 mds1 kernel: uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 Jan 4 16:46:41 mds1 kernel: uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 Jan 4 16:46:41 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 Jan 4 16:46:41 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jan 4 16:46:41 mds1 kernel: usb usb1: Product: UHCI Host Controller Jan 4 16:46:41 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 uhci_hcd Jan 4 16:46:41 mds1 kernel: usb usb1: SerialNumber: 0000:00:01.2 Jan 4 16:46:41 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice Jan 4 16:46:41 mds1 kernel: hub 1-0:1.0: USB hub found Jan 4 16:46:41 mds1 kernel: hub 1-0:1.0: 2 ports detected Jan 4 16:46:41 mds1 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 Jan 4 16:46:41 mds1 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Jan 4 16:46:41 mds1 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Jan 4 16:46:41 mds1 kernel: mice: PS/2 mouse device common for all mice Jan 4 16:46:41 mds1 kernel: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 Jan 4 16:46:41 mds1 kernel: rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 Jan 4 16:46:41 mds1 kernel: rtc0: alarms up to one day, 114 bytes nvram, hpet irqs Jan 4 16:46:41 mds1 kernel: cpuidle: using governor ladder Jan 4 16:46:41 mds1 kernel: cpuidle: using governor menu Jan 4 16:46:41 mds1 kernel: usbcore: registered new interface driver hiddev Jan 4 16:46:41 mds1 kernel: usbcore: registered new interface driver usbhid Jan 4 16:46:41 mds1 kernel: usbhid: v2.6:USB HID core driver Jan 4 16:46:41 mds1 kernel: TCP cubic registered Jan 4 16:46:41 mds1 kernel: Initializing XFRM netlink socket Jan 4 16:46:41 mds1 kernel: NET: Registered protocol family 17 Jan 4 16:46:41 mds1 kernel: registered taskstats version 1 Jan 4 16:46:41 mds1 kernel: rtc_cmos 00:01: setting system clock to 2012-01-04 21:45:55 UTC (1325713555) Jan 4 16:46:41 mds1 kernel: Initalizing network drop monitor service Jan 4 16:46:41 mds1 kernel: Freeing unused kernel memory: 1232k freed Jan 4 16:46:41 mds1 kernel: Write protecting the kernel read-only data: 10240k Jan 4 16:46:41 mds1 kernel: Freeing unused kernel memory: 1108k freed Jan 4 16:46:41 mds1 kernel: Freeing unused kernel memory: 1796k freed Jan 4 16:46:41 mds1 kernel: dracut: dracut-004-32.el6 Jan 4 16:46:41 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation Jan 4 16:46:41 mds1 kernel: device-mapper: uevent: version 1.0.3 Jan 4 16:46:41 mds1 kernel: device-mapper: ioctl: 4.20.6-ioctl (2011-02-02) initialised: dm-devel@redhat.com Jan 4 16:46:41 mds1 kernel: udev: starting version 147 Jan 4 16:46:41 mds1 kernel: dracut: Starting plymouth daemon Jan 4 16:46:41 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation Jan 4 16:46:41 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation Jan 4 16:46:41 mds1 kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 Jan 4 16:46:41 mds1 kernel: scsi0 : ata_piix Jan 4 16:46:41 mds1 kernel: scsi1 : ata_piix Jan 4 16:46:41 mds1 kernel: ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 Jan 4 16:46:41 mds1 kernel: ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10 Jan 4 16:46:41 mds1 kernel: sym53c8xx 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 10 (level, high) -> IRQ 10 Jan 4 16:46:41 mds1 kernel: sym0: <895a> rev 0x0 at pci 0000:00:05.0 irq 10 Jan 4 16:46:41 mds1 kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking Jan 4 16:46:41 mds1 kernel: sym0: SCSI BUS has been reset. Jan 4 16:46:41 mds1 kernel: scsi2 : sym-2.2.3 Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 Jan 4 16:46:41 mds1 kernel: virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 Jan 4 16:46:41 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 Jan 4 16:46:41 mds1 kernel: virtio-pci 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 10 (level, high) -> IRQ 10 Jan 4 16:46:41 mds1 kernel: virtio-pci 0000:00:07.0: PCI INT A -> Link[LNKC] -> GSI 11 (level, high) -> IRQ 11 Jan 4 16:46:41 mds1 kernel: vda: vda1 vda2 Jan 4 16:46:41 mds1 kernel: scsi: waiting for bus probes to complete ... Jan 4 16:46:41 mds1 kernel: sym0: unknown interrupt(s) ignored, ISTAT=0x5 DSTAT=0x80 SIST=0x0 Jan 4 16:46:41 mds1 kernel: scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 0.14 PQ: 0 ANSI: 5 Jan 4 16:46:41 mds1 kernel: scsi target2:0:0: tagged command queuing enabled, command queue depth 16. Jan 4 16:46:41 mds1 kernel: scsi target2:0:0: Beginning Domain Validation Jan 4 16:46:41 mds1 kernel: scsi target2:0:0: Domain Validation skipping write tests Jan 4 16:46:41 mds1 kernel: scsi target2:0:0: Ending Domain Validation Jan 4 16:46:41 mds1 kernel: scsi 2:0:1:0: Direct-Access QEMU QEMU HARDDISK 0.14 PQ: 0 ANSI: 5 Jan 4 16:46:41 mds1 kernel: scsi target2:0:1: tagged command queuing enabled, command queue depth 16. Jan 4 16:46:41 mds1 kernel: scsi target2:0:1: Beginning Domain Validation Jan 4 16:46:41 mds1 kernel: scsi target2:0:1: Domain Validation skipping write tests Jan 4 16:46:41 mds1 kernel: scsi target2:0:1: Ending Domain Validation Jan 4 16:46:41 mds1 kernel: sd 2:0:0:0: [sda] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB) Jan 4 16:46:41 mds1 kernel: sd 2:0:0:0: [sda] Write Protect is off Jan 4 16:46:41 mds1 kernel: sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 4 16:46:41 mds1 kernel: sd 2:0:1:0: [sdb] 204800 512-byte logical blocks: (104 MB/100 MiB) Jan 4 16:46:41 mds1 kernel: sd 2:0:1:0: [sdb] Write Protect is off Jan 4 16:46:41 mds1 kernel: sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 4 16:46:41 mds1 kernel: sda: Jan 4 16:46:41 mds1 kernel: sdb: unknown partition table Jan 4 16:46:41 mds1 kernel: sd 2:0:0:0: [sda] Attached SCSI disk Jan 4 16:46:41 mds1 kernel: unknown partition table Jan 4 16:46:41 mds1 kernel: sd 2:0:1:0: [sdb] Attached SCSI disk Jan 4 16:46:41 mds1 kernel: dracut: Scanning devices vda2 for LVM logical volumes LustreVG/root LustreVG/swap Jan 4 16:46:41 mds1 kernel: dracut: inactive '/dev/LustreVG/root' [1.37 GiB] inherit Jan 4 16:46:41 mds1 kernel: dracut: inactive '/dev/LustreVG/swap' [544.00 MiB] inherit Jan 4 16:46:41 mds1 kernel: kjournald starting. Commit interval 5 seconds Jan 4 16:46:41 mds1 kernel: EXT3-fs (dm-0): mounted filesystem with ordered data mode Jan 4 16:46:41 mds1 kernel: dracut: Mounted root filesystem /dev/mapper/LustreVG-root Jan 4 16:46:41 mds1 kernel: dracut: Loading SELinux policy Jan 4 16:46:41 mds1 kernel: SELinux: Disabled at runtime. Jan 4 16:46:41 mds1 kernel: type=1404 audit(1325713560.751:2): selinux=0 auid=4294967295 ses=4294967295 Jan 4 16:46:41 mds1 kernel: dracut: /sbin/load_policy: Can't load policy: No such file or directory Jan 4 16:46:41 mds1 kernel: dracut: Switching root Jan 4 16:46:41 mds1 kernel: readahead: starting Jan 4 16:46:41 mds1 kernel: udev: starting version 147 Jan 4 16:46:41 mds1 kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 Jan 4 16:46:41 mds1 kernel: Intel ICH 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 Jan 4 16:46:41 mds1 kernel: intel8x0_measure_ac97_clock: measured 50929 usecs (4506 samples) Jan 4 16:46:41 mds1 kernel: intel8x0: measured clock 88476 rejected Jan 4 16:46:41 mds1 kernel: intel8x0_measure_ac97_clock: measured 51051 usecs (4486 samples) Jan 4 16:46:41 mds1 kernel: intel8x0: measured clock 87872 rejected Jan 4 16:46:41 mds1 kernel: intel8x0_measure_ac97_clock: measured 50878 usecs (4505 samples) Jan 4 16:46:41 mds1 kernel: intel8x0: measured clock 88545 rejected Jan 4 16:46:41 mds1 kernel: intel8x0: clocking to 48000 Jan 4 16:46:41 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg0 type 0 Jan 4 16:46:41 mds1 kernel: sd 2:0:1:0: Attached scsi generic sg1 type 0 Jan 4 16:46:41 mds1 kernel: EXT3-fs (dm-0): using internal journal Jan 4 16:46:41 mds1 kernel: kjournald starting. Commit interval 5 seconds Jan 4 16:46:41 mds1 kernel: EXT3-fs (vda1): using internal journal Jan 4 16:46:41 mds1 kernel: EXT3-fs (vda1): mounted filesystem with ordered data mode Jan 4 16:46:41 mds1 kernel: Adding 557048k swap on /dev/mapper/LustreVG-swap. Priority:-1 extents:1 across:557048k Jan 4 16:46:41 mds1 kernel: microcode: CPU0 sig=0x623, pf=0x0, revision=0x0 Jan 4 16:46:41 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-02-03 Jan 4 16:46:41 mds1 kernel: Microcode Update Driver: v2.00 , Peter Oruba Jan 4 16:46:41 mds1 kernel: Microcode Update Driver: v2.00 removed. Jan 4 16:46:41 mds1 kernel: NET: Registered protocol family 10 Jan 4 16:46:41 mds1 kernel: lo: Disabled Privacy Extensions Jan 4 16:46:42 mds1 rpc.statd[1224]: Version 1.2.2 starting Jan 4 16:46:42 mds1 sm-notify[1225]: Version 1.2.2 starting Jan 4 16:46:43 mds1 kernel: RPC: Registered udp transport module. Jan 4 16:46:43 mds1 kernel: RPC: Registered tcp transport module. Jan 4 16:46:43 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Jan 4 16:46:43 mds1 corosync[1290]: [MAIN ] Corosync Cluster Engine ('1.2.3'): started and ready to provide service. Jan 4 16:46:43 mds1 corosync[1290]: [MAIN ] Corosync built-in features: nss rdma Jan 4 16:46:43 mds1 corosync[1290]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'. Jan 4 16:46:43 mds1 corosync[1290]: [TOTEM ] Initializing transport (UDP/IP). Jan 4 16:46:43 mds1 corosync[1290]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0). Jan 4 16:46:43 mds1 corosync[1290]: [TOTEM ] The network interface [192.168.122.89] is now up. Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: process_ais_conf: Reading configure Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: config_find_init: Local handle: 2013064636357672962 for logging Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: config_find_next: Processing additional logging options... Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: Found 'off' for option: debug Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: Found 'no' for option: to_logfile Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: config_find_init: Local handle: 4730966301143465987 for quorum Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: config_find_next: No additional configuration supplied for: quorum Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: No default for option: provider Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: config_find_init: Local handle: 7739444317642555396 for service Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: config_find_next: Processing additional service options... Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: Defaulting to '0' for option: ver Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_startup: CRM: Initialized Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] Logging: Initialized pcmk_startup Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615 Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_startup: Service: 10 Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_startup: Local hostname: mds1 Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_update_nodeid: Local node id: 1501210816 Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: update_member: Creating entry for node 1501210816 born on 0 Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: update_member: 0x24221c0 Node 1501210816 now known as mds1 (was: (null)) Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: update_member: Node mds1 now has 1 quorum votes (was 0) Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: update_member: Node 1501210816/mds1 is now: member Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: spawn_child: Forked child 1300 for process stonith-ng Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: spawn_child: Forked child 1301 for process cib Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: spawn_child: Forked child 1302 for process lrmd Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: spawn_child: Forked child 1303 for process attrd Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: spawn_child: Forked child 1304 for process pengine Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: spawn_child: Forked child 1305 for process crmd Jan 4 16:46:44 mds1 corosync[1290]: [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.2 Jan 4 16:46:44 mds1 corosync[1290]: [SERV ] Service engine loaded: corosync extended virtual synchrony service Jan 4 16:46:44 mds1 corosync[1290]: [SERV ] Service engine loaded: corosync configuration service Jan 4 16:46:44 mds1 corosync[1290]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 Jan 4 16:46:44 mds1 corosync[1290]: [SERV ] Service engine loaded: corosync cluster config database access v1.01 Jan 4 16:46:44 mds1 corosync[1290]: [SERV ] Service engine loaded: corosync profile loading service Jan 4 16:46:44 mds1 corosync[1290]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 Jan 4 16:46:44 mds1 corosync[1290]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine. Jan 4 16:46:44 mds1 corosync[1290]: [TOTEM ] Process pause detected for 629 ms, flushing membership messages. Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: Invoked: /usr/lib64/heartbeat/stonithd Jan 4 16:46:44 mds1 cib: [1301]: info: Invoked: /usr/lib64/heartbeat/cib Jan 4 16:46:44 mds1 lrmd: [1302]: info: G_main_add_SignalHandler: Added signal handler for signal 15 Jan 4 16:46:44 mds1 lrmd: [1302]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Jan 4 16:46:44 mds1 lrmd: [1302]: info: enabling coredumps Jan 4 16:46:44 mds1 attrd: [1303]: info: Invoked: /usr/lib64/heartbeat/attrd Jan 4 16:46:44 mds1 pengine: [1304]: info: Invoked: /usr/lib64/heartbeat/pengine Jan 4 16:46:44 mds1 crmd: [1305]: info: Invoked: /usr/lib64/heartbeat/crmd Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/root Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: get_cluster_type: Cluster type is: 'openais'. Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5032: memb=0, new=0, lost=0 Jan 4 16:46:44 mds1 cib: [1301]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Jan 4 16:46:44 mds1 lrmd: [1302]: info: G_main_add_SignalHandler: Added signal handler for signal 10 Jan 4 16:46:44 mds1 pengine: [1304]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Jan 4 16:46:44 mds1 crmd: [1305]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Jan 4 16:46:44 mds1 attrd: [1303]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5032: memb=1, new=1, lost=0 Jan 4 16:46:44 mds1 cib: [1301]: info: G_main_add_TriggerHandler: Added signal manual handler Jan 4 16:46:44 mds1 lrmd: [1302]: info: G_main_add_SignalHandler: Added signal handler for signal 12 Jan 4 16:46:44 mds1 crmd: [1305]: info: main: CRM Hg Version: f059ec7ced7a86f18e5490b67ebf4a0b963bccfe Jan 4 16:46:44 mds1 attrd: [1303]: info: main: Starting up Jan 4 16:46:44 mds1 pengine: [1304]: info: main: Starting pengine Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: init_ais_connection_classic: AIS connection established Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: NEW: mds1 1501210816 Jan 4 16:46:44 mds1 cib: [1301]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Jan 4 16:46:44 mds1 lrmd: [1302]: info: Started. Jan 4 16:46:44 mds1 crmd: [1305]: info: crmd_init: Starting crmd Jan 4 16:46:44 mds1 attrd: [1303]: info: get_cluster_type: Cluster type is: 'openais'. Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: get_ais_nodeid: Server details: id=1501210816 uname=mds1 cname=pcmk Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 16:46:44 mds1 crmd: [1305]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Jan 4 16:46:44 mds1 attrd: [1303]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: update_member: Node mds1 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 16:46:44 mds1 attrd: [1303]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: crm_new_peer: Node mds1 now has id: 1501210816 Jan 4 16:46:44 mds1 corosync[1290]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 16:46:44 mds1 attrd: [1303]: info: init_ais_connection_classic: AIS connection established Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: crm_new_peer: Node 1501210816 is now known as mds1 Jan 4 16:46:44 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:44 mds1 attrd: [1303]: info: get_ais_nodeid: Server details: id=1501210816 uname=mds1 cname=pcmk Jan 4 16:46:44 mds1 stonith-ng: [1300]: info: main: Starting stonith-ng mainloop Jan 4 16:46:44 mds1 corosync[1290]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 16:46:44 mds1 attrd: [1303]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Jan 4 16:46:44 mds1 corosync[1290]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 16:46:44 mds1 attrd: [1303]: info: crm_new_peer: Node mds1 now has id: 1501210816 Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc: Recorded connection 0x242b4c0 for stonith-ng/1300 Jan 4 16:46:44 mds1 attrd: [1303]: info: crm_new_peer: Node 1501210816 is now known as mds1 Jan 4 16:46:44 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc: Recorded connection 0x242fc90 for attrd/1303 Jan 4 16:46:44 mds1 attrd: [1303]: info: main: Cluster connection active Jan 4 16:46:44 mds1 attrd: [1303]: info: main: Accepting attribute updates Jan 4 16:46:44 mds1 attrd: [1303]: info: main: Starting mainloop... Jan 4 16:46:44 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) Jan 4 16:46:45 mds1 crmd: [1305]: info: do_cib_control: Could not connect to the CIB service: connection failed Jan 4 16:46:45 mds1 crmd: [1305]: WARN: do_cib_control: Couldn't complete CIB registration 1 times... pause and retry Jan 4 16:46:45 mds1 crmd: [1305]: info: crmd_init: Starting crmd's mainloop Jan 4 16:46:46 mds1 avahi-daemon[1330]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70). Jan 4 16:46:46 mds1 avahi-daemon[1330]: Successfully dropped root privileges. Jan 4 16:46:46 mds1 avahi-daemon[1330]: avahi-daemon 0.6.25 starting up. Jan 4 16:46:47 mds1 crmd: [1305]: info: crm_timer_popped: Wait Timer (I_NULL) just popped! Jan 4 16:46:47 mds1 avahi-daemon[1330]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Jan 4 16:46:47 mds1 avahi-daemon[1330]: Successfully called chroot(). Jan 4 16:46:47 mds1 avahi-daemon[1330]: Successfully dropped remaining capabilities. Jan 4 16:46:47 mds1 avahi-daemon[1330]: Loading service file /services/ssh.service. Jan 4 16:46:47 mds1 avahi-daemon[1330]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.122.89. Jan 4 16:46:47 mds1 avahi-daemon[1330]: New relevant interface eth0.IPv4 for mDNS. Jan 4 16:46:47 mds1 avahi-daemon[1330]: Network interface enumeration completed. Jan 4 16:46:47 mds1 avahi-daemon[1330]: Registering new address record for fe80::5054:ff:fed9:98ec on eth0.*. Jan 4 16:46:47 mds1 avahi-daemon[1330]: Registering new address record for 192.168.122.89 on eth0.IPv4. Jan 4 16:46:47 mds1 avahi-daemon[1330]: Registering HINFO record with values 'X86_64'/'LINUX'. Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5040: memb=1, new=0, lost=0 Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5040: memb=2, new=1, lost=0 Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: update_member: Creating entry for node 2608507072 born on 5040 Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: update_member: Node 2608507072/unknown is now: member Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2608507072 Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2608507072 Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: send_member_notification: Sending membership update 5040 to 0 children Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: update_member: 0x24221c0 Node 1501210816 ((null)) born on: 5040 Jan 4 16:46:47 mds1 corosync[1290]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: update_member: 0x23f3e30 Node 2608507072 (mds2) born on: 5040 Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: update_member: 0x23f3e30 Node 2608507072 now known as mds2 (was: (null)) Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: update_member: Node mds2 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: update_member: Node mds2 now has 1 quorum votes (was 0) Jan 4 16:46:47 mds1 corosync[1290]: [pcmk ] info: send_member_notification: Sending membership update 5040 to 0 children Jan 4 16:46:47 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:47 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:47 mds1 corosync[1290]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 16:46:47 mds1 corosync[1290]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 16:46:47 mds1 cib: [1301]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-79.raw Jan 4 16:46:48 mds1 avahi-daemon[1330]: Server startup complete. Host name is mds1.local. Local service cookie is 2722282696. Jan 4 16:46:48 mds1 crmd: [1305]: info: do_cib_control: Could not connect to the CIB service: connection failed Jan 4 16:46:48 mds1 crmd: [1305]: WARN: do_cib_control: Couldn't complete CIB registration 2 times... pause and retry Jan 4 16:46:49 mds1 cib: [1301]: info: write_cib_contents: Wrote version 0.246.0 of the CIB to disk (digest: 18bcf148b34d1cf3a765559cad18447e) Jan 4 16:46:49 mds1 kernel: TECH PREVIEW: fscache may not be fully supported. Jan 4 16:46:49 mds1 kernel: Please review provided documentation for limitations. Jan 4 16:46:49 mds1 kernel: Disabling lock debugging due to kernel taint Jan 4 16:46:49 mds1 kernel: Slow work thread pool: Starting up Jan 4 16:46:49 mds1 kernel: Slow work thread pool: Ready Jan 4 16:46:49 mds1 kernel: FS-Cache: Loaded Jan 4 16:46:49 mds1 avahi-daemon[1330]: Service "mds1" (/services/ssh.service) successfully established. Jan 4 16:46:49 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.05e3TV (digest: /var/lib/heartbeat/crm/cib.lZ8SKH) Jan 4 16:46:49 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching Jan 4 16:46:49 mds1 acpid: starting up Jan 4 16:46:49 mds1 acpid: 1 rule loaded Jan 4 16:46:49 mds1 acpid: waiting for events: event logging is off Jan 4 16:46:50 mds1 cib: [1301]: info: startCib: CIB Initialization completed successfully Jan 4 16:46:50 mds1 cib: [1301]: info: get_cluster_type: Cluster type is: 'openais'. Jan 4 16:46:50 mds1 cib: [1301]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Jan 4 16:46:50 mds1 cib: [1301]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Jan 4 16:46:50 mds1 cib: [1301]: info: init_ais_connection_classic: AIS connection established Jan 4 16:46:50 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc: Recorded connection 0x2433ff0 for cib/1301 Jan 4 16:46:50 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc: Sending membership update 5040 to cib Jan 4 16:46:50 mds1 cib: [1301]: info: get_ais_nodeid: Server details: id=1501210816 uname=mds1 cname=pcmk Jan 4 16:46:50 mds1 cib: [1301]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Jan 4 16:46:50 mds1 cib: [1301]: info: crm_new_peer: Node mds1 now has id: 1501210816 Jan 4 16:46:50 mds1 cib: [1301]: info: crm_new_peer: Node 1501210816 is now known as mds1 Jan 4 16:46:50 mds1 cib: [1301]: info: cib_init: Starting cib mainloop Jan 4 16:46:50 mds1 cib: [1301]: notice: ais_dispatch_message: Membership 5040: quorum acquired Jan 4 16:46:50 mds1 cib: [1301]: info: crm_update_peer: Node mds1: id=1501210816 state=member (new) addr=r(0) ip(192.168.122.89) (new) votes=1 (new) born=5040 seen=5040 proc=00000000000000000000000000111312 (new) Jan 4 16:46:50 mds1 cib: [1301]: info: crm_new_peer: Node mds2 now has id: 2608507072 Jan 4 16:46:50 mds1 cib: [1301]: info: crm_new_peer: Node 2608507072 is now known as mds2 Jan 4 16:46:50 mds1 cib: [1301]: info: crm_update_peer: Node mds2: id=2608507072 state=member (new) addr=r(0) ip(192.168.122.155) votes=1 born=5040 seen=5040 proc=00000000000000000000000000111312 Jan 4 16:46:50 mds1 crmd: [1305]: info: crm_timer_popped: Wait Timer (I_NULL) just popped! Jan 4 16:46:50 mds1 crmd: [1305]: info: do_cib_control: CIB connection established Jan 4 16:46:50 mds1 crmd: [1305]: info: get_cluster_type: Cluster type is: 'openais'. Jan 4 16:46:50 mds1 crmd: [1305]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Jan 4 16:46:50 mds1 crmd: [1305]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Jan 4 16:46:50 mds1 crmd: [1305]: info: init_ais_connection_classic: AIS connection established Jan 4 16:46:50 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc: Recorded connection 0x2438350 for crmd/1305 Jan 4 16:46:50 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc: Sending membership update 5040 to crmd Jan 4 16:46:50 mds1 crmd: [1305]: info: get_ais_nodeid: Server details: id=1501210816 uname=mds1 cname=pcmk Jan 4 16:46:50 mds1 crmd: [1305]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Jan 4 16:46:50 mds1 crmd: [1305]: info: crm_new_peer: Node mds1 now has id: 1501210816 Jan 4 16:46:50 mds1 crmd: [1305]: info: crm_new_peer: Node 1501210816 is now known as mds1 Jan 4 16:46:50 mds1 crmd: [1305]: info: ais_status_callback: status: mds1 is now unknown Jan 4 16:46:50 mds1 crmd: [1305]: info: do_ha_control: Connected to the cluster Jan 4 16:46:50 mds1 crmd: [1305]: info: do_started: Delaying start, no membership data (0000000000100000) Jan 4 16:46:50 mds1 crmd: [1305]: notice: ais_dispatch_message: Membership 5040: quorum acquired Jan 4 16:46:50 mds1 crmd: [1305]: notice: crmd_peer_update: Status update: Client mds1/crmd now has status [online] (DC=) Jan 4 16:46:50 mds1 crmd: [1305]: info: ais_status_callback: status: mds1 is now member (was unknown) Jan 4 16:46:50 mds1 crmd: [1305]: info: crm_update_peer: Node mds1: id=1501210816 state=member (new) addr=r(0) ip(192.168.122.89) (new) votes=1 (new) born=5040 seen=5040 proc=00000000000000000000000000111312 (new) Jan 4 16:46:50 mds1 crmd: [1305]: info: crm_new_peer: Node mds2 now has id: 2608507072 Jan 4 16:46:50 mds1 crmd: [1305]: info: crm_new_peer: Node 2608507072 is now known as mds2 Jan 4 16:46:50 mds1 crmd: [1305]: info: ais_status_callback: status: mds2 is now unknown Jan 4 16:46:50 mds1 crmd: [1305]: info: ais_status_callback: status: mds2 is now member (was unknown) Jan 4 16:46:50 mds1 crmd: [1305]: info: crm_update_peer: Node mds2: id=2608507072 state=member (new) addr=r(0) ip(192.168.122.155) votes=1 born=5040 seen=5040 proc=00000000000000000000000000111312 Jan 4 16:46:50 mds1 crmd: [1305]: info: do_started: Delaying start, Config not read (0000000000000040) Jan 4 16:46:50 mds1 crmd: [1305]: info: config_query_callback: Checking for expired actions every 900000ms Jan 4 16:46:50 mds1 crmd: [1305]: info: config_query_callback: Sending expected-votes=4 to corosync Jan 4 16:46:50 mds1 crmd: [1305]: info: do_started: The local CRM is operational Jan 4 16:46:50 mds1 corosync[1290]: [pcmk ] info: update_expected_votes: Expected quorum votes 2 -> 4 Jan 4 16:46:50 mds1 crmd: [1305]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ] Jan 4 16:46:50 mds1 acpid: client connected from 1414[68:68] Jan 4 16:46:50 mds1 acpid: 1 client rule loaded Jan 4 16:46:51 mds1 crmd: [1305]: notice: ais_dispatch_message: Membership 5040: quorum lost Jan 4 16:46:51 mds1 crmd: [1305]: info: te_connect_stonith: Attempting connection to fencing daemon... Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5048: memb=2, new=0, lost=0 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: memb: mds2 2608507072 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5048: memb=3, new=1, lost=0 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: update_member: Creating entry for node 2692393152 born on 5048 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: update_member: Node 2692393152/unknown is now: member Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2692393152 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: mds2 2608507072 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2692393152 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: send_member_notification: Sending membership update 5048 to 2 children Jan 4 16:46:51 mds1 corosync[1290]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 16:46:51 mds1 cib: [1301]: notice: ais_dispatch_message: Membership 5048: quorum lost Jan 4 16:46:51 mds1 cib: [1301]: info: crm_new_peer: Node now has id: 2692393152 Jan 4 16:46:51 mds1 cib: [1301]: info: crm_update_peer: Node (null): id=2692393152 state=member (new) addr=r(0) ip(192.168.122.160) votes=0 born=0 seen=5048 proc=00000000000000000000000000000000 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: update_member: 0x242a790 Node 2692393152 (oss2) born on: 5048 Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: update_member: 0x242a790 Node 2692393152 now known as oss2 (was: (null)) Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: update_member: Node oss2 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: update_member: Node oss2 now has 1 quorum votes (was 0) Jan 4 16:46:51 mds1 corosync[1290]: [pcmk ] info: send_member_notification: Sending membership update 5048 to 2 children Jan 4 16:46:51 mds1 cib: [1301]: notice: ais_dispatch_message: Membership 5048: quorum acquired Jan 4 16:46:51 mds1 cib: [1301]: info: crm_get_peer: Node 2692393152 is now known as oss2 Jan 4 16:46:51 mds1 cib: [1301]: info: crm_update_peer: Node oss2: id=2692393152 state=member addr=r(0) ip(192.168.122.160) votes=1 (new) born=5048 seen=5048 proc=00000000000000000000000000111312 (new) Jan 4 16:46:51 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:51 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:51 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:51 mds1 corosync[1290]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 16:46:51 mds1 corosync[1290]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 16:46:52 mds1 crmd: [1305]: info: te_connect_stonith: Connected Jan 4 16:46:52 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5048: quorum still lost Jan 4 16:46:52 mds1 crmd: [1305]: info: crm_new_peer: Node now has id: 2692393152 Jan 4 16:46:52 mds1 crmd: [1305]: info: crm_update_peer: Node (null): id=2692393152 state=member (new) addr=r(0) ip(192.168.122.160) votes=0 born=0 seen=5048 proc=00000000000000000000000000000000 Jan 4 16:46:52 mds1 crmd: [1305]: notice: ais_dispatch_message: Membership 5048: quorum acquired Jan 4 16:46:52 mds1 crmd: [1305]: info: crm_get_peer: Node 2692393152 is now known as oss2 Jan 4 16:46:52 mds1 crmd: [1305]: info: ais_status_callback: status: oss2 is now member Jan 4 16:46:52 mds1 crmd: [1305]: notice: crmd_peer_update: Status update: Client oss2/crmd now has status [online] (DC=) Jan 4 16:46:52 mds1 crmd: [1305]: info: crm_update_peer: Node oss2: id=2692393152 state=member addr=r(0) ip(192.168.122.160) votes=1 (new) born=5048 seen=5048 proc=00000000000000000000000000111312 (new) Jan 4 16:46:54 mds1 attrd: [1303]: info: cib_connect: Connected to the CIB after 2 signon attempts Jan 4 16:46:54 mds1 attrd: [1303]: info: cib_connect: Sending full refresh Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5052: memb=3, new=0, lost=0 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: memb: mds2 2608507072 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: memb: oss2 2692393152 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5052: memb=4, new=1, lost=0 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: update_member: Creating entry for node 2474289344 born on 5052 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: update_member: Node 2474289344/unknown is now: member Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2474289344 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2474289344 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: mds2 2608507072 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: pcmk_peer_update: MEMB: oss2 2692393152 Jan 4 16:46:56 mds1 cib: [1301]: info: ais_dispatch_message: Membership 5052: quorum retained Jan 4 16:46:56 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5052: quorum retained Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: send_member_notification: Sending membership update 5052 to 2 children Jan 4 16:46:56 mds1 cib: [1301]: info: crm_new_peer: Node now has id: 2474289344 Jan 4 16:46:56 mds1 crmd: [1305]: info: crm_new_peer: Node now has id: 2474289344 Jan 4 16:46:56 mds1 corosync[1290]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 16:46:56 mds1 cib: [1301]: info: crm_update_peer: Node (null): id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=0 born=0 seen=5052 proc=00000000000000000000000000000000 Jan 4 16:46:56 mds1 crmd: [1305]: info: crm_update_peer: Node (null): id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=0 born=0 seen=5052 proc=00000000000000000000000000000000 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: update_member: 0x242a820 Node 2474289344 (oss1) born on: 5052 Jan 4 16:46:56 mds1 cib: [1301]: info: ais_dispatch_message: Membership 5052: quorum retained Jan 4 16:46:56 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5052: quorum retained Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: update_member: 0x242a820 Node 2474289344 now known as oss1 (was: (null)) Jan 4 16:46:56 mds1 cib: [1301]: info: crm_get_peer: Node 2474289344 is now known as oss1 Jan 4 16:46:56 mds1 crmd: [1305]: info: crm_get_peer: Node 2474289344 is now known as oss1 Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: update_member: Node oss1 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 16:46:56 mds1 cib: [1301]: info: crm_update_peer: Node oss1: id=2474289344 state=member addr=r(0) ip(192.168.122.147) votes=1 (new) born=5052 seen=5052 proc=00000000000000000000000000111312 (new) Jan 4 16:46:56 mds1 crmd: [1305]: info: ais_status_callback: status: oss1 is now member Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: update_member: Node oss1 now has 1 quorum votes (was 0) Jan 4 16:46:56 mds1 crmd: [1305]: notice: crmd_peer_update: Status update: Client oss1/crmd now has status [online] (DC=) Jan 4 16:46:56 mds1 corosync[1290]: [pcmk ] info: send_member_notification: Sending membership update 5052 to 2 children Jan 4 16:46:56 mds1 crmd: [1305]: info: crm_update_peer: Node oss1: id=2474289344 state=member addr=r(0) ip(192.168.122.147) votes=1 (new) born=5052 seen=5052 proc=00000000000000000000000000111312 (new) Jan 4 16:46:56 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:56 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:56 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:56 mds1 corosync[1290]: [CPG ] downlist received left_list: 0 Jan 4 16:46:56 mds1 corosync[1290]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 16:46:56 mds1 corosync[1290]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 16:46:57 mds1 avahi-dnsconfd[1557]: Successfully connected to Avahi daemon. Jan 4 16:47:51 mds1 crmd: [1305]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped! Jan 4 16:47:51 mds1 crmd: [1305]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING Jan 4 16:47:51 mds1 crmd: [1305]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 16:47:51 mds1 crmd: [1305]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Jan 4 16:47:51 mds1 crmd: [1305]: info: do_te_control: Registering TE UUID: b4a0159a-ec35-4987-a4c9-c832425ca02f Jan 4 16:47:51 mds1 crmd: [1305]: WARN: cib_client_add_notify_callback: Callback already present Jan 4 16:47:51 mds1 crmd: [1305]: info: set_graph_functions: Setting custom graph functions Jan 4 16:47:51 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses Jan 4 16:47:51 mds1 crmd: [1305]: info: do_dc_takeover: Taking over DC status for this partition Jan 4 16:47:51 mds1 cib: [1301]: info: cib_process_readwrite: We are now in R/W mode Jan 4 16:47:51 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/6, version=0.246.0): ok (rc=0) Jan 4 16:47:51 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/7, version=0.246.0): ok (rc=0) Jan 4 16:47:51 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/9, version=0.246.0): ok (rc=0) Jan 4 16:47:51 mds1 crmd: [1305]: info: join_make_offer: Making join offers based on membership 5052 Jan 4 16:47:51 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/11, version=0.246.0): ok (rc=0) Jan 4 16:47:51 mds1 crmd: [1305]: info: do_dc_join_offer_all: join-1: Waiting on 4 outstanding join acks Jan 4 16:47:51 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5052: quorum retained Jan 4 16:47:51 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 16:47:51 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/14, version=0.246.0): ok (rc=0) Jan 4 16:47:51 mds1 crmd: [1305]: info: config_query_callback: Checking for expired actions every 900000ms Jan 4 16:47:51 mds1 crmd: [1305]: info: config_query_callback: Sending expected-votes=4 to corosync Jan 4 16:47:51 mds1 crmd: [1305]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 16:47:51 mds1 crmd: [1305]: info: ais_dispatch_message: Membership 5052: quorum retained Jan 4 16:47:51 mds1 crmd: [1305]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 16:47:51 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/17, version=0.246.0): ok (rc=0) Jan 4 16:47:51 mds1 crmd: [1305]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 16:47:51 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes responded to the join offer. Jan 4 16:47:51 mds1 crmd: [1305]: info: do_dc_join_finalize: join-1: Syncing the CIB from oss1 to the rest of the cluster Jan 4 16:47:51 mds1 cib: [1301]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-80.raw Jan 4 16:47:51 mds1 cib: [1301]: info: write_cib_contents: Wrote version 0.247.0 of the CIB to disk (digest: 4ea50f110efd2821cf0f5a8651199597) Jan 4 16:47:51 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.WCEfjD (digest: /var/lib/heartbeat/crm/cib.NAqQoc) Jan 4 16:47:51 mds1 cib: [1301]: info: cib_replace_notify: Local-only Replace: -1.-1.-1 from oss1 Jan 4 16:47:51 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 16:47:52 mds1 cib: [1301]: info: log_data_element: cib:diff: + Jan 4 16:47:52 mds1 attrd: [1303]: info: do_cib_replaced: Sending full refresh Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/19, version=0.247.0): ok (rc=0) Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/20, version=0.247.0): ok (rc=0) Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/21, version=0.247.0): ok (rc=0) Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/22, version=0.247.0): ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_dc_join_ack: join-1: Updating node state to member for oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: do_dc_join_ack: join-1: Updating node state to member for oss2 Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/transient_attributes (origin=oss1/crmd/7, version=0.247.0): ok (rc=0) Jan 4 16:47:52 mds1 attrd: [1303]: info: crm_new_peer: Node oss1 now has id: 2474289344 Jan 4 16:47:52 mds1 crmd: [1305]: info: update_attrd: Connecting to attrd... Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss2']/transient_attributes (origin=oss2/crmd/7, version=0.247.0): ok (rc=0) Jan 4 16:47:52 mds1 attrd: [1303]: info: crm_new_peer: Node 2474289344 is now known as oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: do_dc_join_ack: join-1: Updating node state to member for mds2 Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds2']/transient_attributes (origin=mds2/crmd/7, version=0.247.0): ok (rc=0) Jan 4 16:47:52 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for shutdown Jan 4 16:47:52 mds1 crmd: [1305]: info: do_dc_join_ack: join-1: Updating node state to member for mds1 Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/lrm (origin=local/crmd/23, version=0.247.0): ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss1']/lrm": ok (rc=0) Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss2']/lrm (origin=local/crmd/25, version=0.247.1): ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds1']/transient_attributes (origin=local/crmd/27, version=0.247.2): ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds2']/lrm (origin=local/crmd/28, version=0.247.2): ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds1']/lrm (origin=local/crmd/30, version=0.247.3): ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: crm_update_quorum: Updating quorum status to true (call=34) Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/32, version=0.247.4): ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: abort_transition_graph: do_te_invoke:176 - Triggered transition abort (complete=1) : Peer Cancelled Jan 4 16:47:52 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for terminate Jan 4 16:47:52 mds1 crmd: [1305]: info: do_pe_invoke: Query 35: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:47:52 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss2']/lrm": ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='mds1']/transient_attributes": ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='mds2']/lrm": ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: erase_xpath_callback: Deletion of "//node_state[@uname='mds1']/lrm": ok (rc=0) Jan 4 16:47:52 mds1 cib: [1301]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-81.raw Jan 4 16:47:52 mds1 cib: [1301]: info: write_cib_contents: Wrote version 0.248.0 of the CIB to disk (digest: e53cbf2a880b539ff2813f049dd4c098) Jan 4 16:47:52 mds1 cib: [1301]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.UhEZfM (digest: /var/lib/heartbeat/crm/cib.rw986l) Jan 4 16:47:52 mds1 cib: [1301]: info: log_data_element: cib:diff: - Jan 4 16:47:52 mds1 cib: [1301]: info: log_data_element: cib:diff: + Jan 4 16:47:52 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/34, version=0.248.1): ok (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change Jan 4 16:47:52 mds1 crmd: [1305]: info: need_abort: Aborting on change to admin_epoch Jan 4 16:47:52 mds1 crmd: [1305]: info: do_pe_invoke: Query 36: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:47:52 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=36, ref=pe_calc-dc-1325713672-17, seq=5052, quorate=1 Jan 4 16:47:52 mds1 attrd: [1303]: info: crm_new_peer: Node oss2 now has id: 2692393152 Jan 4 16:47:52 mds1 attrd: [1303]: info: crm_new_peer: Node 2692393152 is now known as oss2 Jan 4 16:47:52 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 16:47:52 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 16:47:52 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 16:47:52 mds1 attrd: [1303]: info: crm_new_peer: Node mds2 now has id: 2608507072 Jan 4 16:47:52 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 16:47:52 mds1 attrd: [1303]: info: crm_new_peer: Node 2608507072 is now known as mds2 Jan 4 16:47:52 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 16:47:52 mds1 attrd: [1303]: info: attrd_local_callback: Sending full refresh (origin=crmd) Jan 4 16:47:52 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 16:47:52 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Jan 4 16:47:52 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 16:47:52 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Jan 4 16:47:52 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Stopped Jan 4 16:47:52 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Stopped Jan 4 16:47:52 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Stopped Jan 4 16:47:52 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Stopped Jan 4 16:47:52 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 16:47:52 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 16:47:52 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 16:47:52 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 16:47:52 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 16:47:52 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 16:47:52 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose mds2. Jan 4 16:47:52 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 16:47:52 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 16:47:52 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 16:47:52 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 16:47:52 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 16:47:52 mds1 pengine: [1304]: notice: LogActions: Start st-oss2#011(oss1) Jan 4 16:47:52 mds1 pengine: [1304]: notice: LogActions: Start st-oss1#011(oss2) Jan 4 16:47:52 mds1 pengine: [1304]: notice: LogActions: Start st-mds1#011(mds2) Jan 4 16:47:52 mds1 pengine: [1304]: notice: LogActions: Start st-mds2#011(mds1) Jan 4 16:47:52 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 16:47:52 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 16:47:52 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(oss1) Jan 4 16:47:52 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(oss1) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 16:47:52 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 0: 49 actions in 49 synapses Jan 4 16:47:52 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1325713672-17) derived from /var/lib/pengine/pe-input-40.bz2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 4: monitor st-oss2_monitor_0 on oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 13: monitor st-oss2_monitor_0 on oss2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 22: monitor st-oss2_monitor_0 on mds2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 31: monitor st-oss2_monitor_0 on mds1 (local) Jan 4 16:47:52 mds1 lrmd: [1302]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Jan 4 16:47:52 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=31:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f op=st-oss2_monitor_0 ) Jan 4 16:47:52 mds1 lrmd: [1302]: info: rsc:st-oss2:2: probe Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 5: monitor st-oss1_monitor_0 on oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 14: monitor st-oss1_monitor_0 on oss2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 23: monitor st-oss1_monitor_0 on mds2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 32: monitor st-oss1_monitor_0 on mds1 (local) Jan 4 16:47:52 mds1 lrmd: [1302]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Jan 4 16:47:52 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=32:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f op=st-oss1_monitor_0 ) Jan 4 16:47:52 mds1 lrmd: [1302]: info: rsc:st-oss1:3: probe Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 6: monitor st-mds1_monitor_0 on oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 15: monitor st-mds1_monitor_0 on oss2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 24: monitor st-mds1_monitor_0 on mds2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 33: monitor st-mds1_monitor_0 on mds1 (local) Jan 4 16:47:52 mds1 lrmd: [1302]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Jan 4 16:47:52 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=33:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f op=st-mds1_monitor_0 ) Jan 4 16:47:52 mds1 lrmd: [1302]: info: rsc:st-mds1:4: probe Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 7: monitor st-mds2_monitor_0 on oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 16: monitor st-mds2_monitor_0 on oss2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 25: monitor st-mds2_monitor_0 on mds2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 34: monitor st-mds2_monitor_0 on mds1 (local) Jan 4 16:47:52 mds1 lrmd: [1302]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Jan 4 16:47:52 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=34:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f op=st-mds2_monitor_0 ) Jan 4 16:47:52 mds1 lrmd: [1302]: info: rsc:st-mds2:5: probe Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 8: monitor MGS_1_monitor_0 on oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 17: monitor MGS_1_monitor_0 on oss2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 26: monitor MGS_1_monitor_0 on mds2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 35: monitor MGS_1_monitor_0 on mds1 (local) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=35:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f op=MGS_1_monitor_0 ) Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 9: monitor lustre-MDT0000_2_monitor_0 on oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 18: monitor lustre-MDT0000_2_monitor_0 on oss2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 27: monitor lustre-MDT0000_2_monitor_0 on mds2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 36: monitor lustre-MDT0000_2_monitor_0 on mds1 (local) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=36:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f op=lustre-MDT0000_2_monitor_0 ) Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 10: monitor lustre-OST0001_4_monitor_0 on oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 19: monitor lustre-OST0001_4_monitor_0 on oss2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 28: monitor lustre-OST0001_4_monitor_0 on mds2 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 37: monitor lustre-OST0001_4_monitor_0 on mds1 (local) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=37:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f op=lustre-OST0001_4_monitor_0 ) Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 11: monitor lustre-OST0000_3_monitor_0 on oss1 Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 20: monitor lustre-OST0000_3_monitor_0 on oss2 Jan 4 16:47:52 mds1 stonith-ng: [1300]: notice: stonith_device_action: Device st-oss1 not found Jan 4 16:47:52 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_execute from lrmd: rc=-11 Jan 4 16:47:52 mds1 stonith-ng: [1300]: notice: stonith_device_action: Device st-mds2 not found Jan 4 16:47:52 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_execute from lrmd: rc=-11 Jan 4 16:47:52 mds1 stonith-ng: [1300]: notice: stonith_device_action: Device st-mds1 not found Jan 4 16:47:52 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_execute from lrmd: rc=-11 Jan 4 16:47:52 mds1 stonith-ng: [1300]: notice: stonith_device_action: Device st-oss2 not found Jan 4 16:47:52 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_execute from lrmd: rc=-11 Jan 4 16:47:52 mds1 crmd: [1305]: info: process_lrm_event: LRM operation st-oss1_monitor_0 (call=3, rc=7, cib-update=37, confirmed=true) not running Jan 4 16:47:52 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_monitor_0 (32) confirmed on mds1 (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 29: monitor lustre-OST0000_3_monitor_0 on mds2 Jan 4 16:47:52 mds1 crmd: [1305]: info: process_lrm_event: LRM operation st-mds1_monitor_0 (call=4, rc=7, cib-update=38, confirmed=true) not running Jan 4 16:47:52 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_monitor_0 (33) confirmed on mds1 (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 38: monitor lustre-OST0000_3_monitor_0 on mds1 (local) Jan 4 16:47:52 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=38:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f op=lustre-OST0000_3_monitor_0 ) Jan 4 16:47:52 mds1 lrmd: [1302]: info: rsc:lustre-OST0000_3:9: probe Jan 4 16:47:52 mds1 crmd: [1305]: info: process_lrm_event: LRM operation st-mds2_monitor_0 (call=5, rc=7, cib-update=39, confirmed=true) not running Jan 4 16:47:52 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_monitor_0 (34) confirmed on mds1 (rc=0) Jan 4 16:47:52 mds1 crmd: [1305]: info: process_lrm_event: LRM operation st-oss2_monitor_0 (call=2, rc=7, cib-update=40, confirmed=true) not running Jan 4 16:47:52 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_monitor_0 (31) confirmed on mds1 (rc=0) Jan 4 16:47:52 mds1 lrmd: [1302]: info: RA output: (lustre-OST0000_3:probe:stderr) Traceback (most recent call last):#012#012 File "/usr/bin/hydra-agent.py", line 144, in #012 result = args.func(args)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py", line 315, in target_running#012 info = store_get_target_info(args.uuid)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/store.py", line 26, in store_get_target_info#012 raise RuntimeError("Failed to read target data for '%s', is it configured? (%s)" % (name, e))#012#012RuntimeError: Failed to read target data for '1a698033-3abc-41e5-a278-d6f06e170348', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/1a698033-3abc-41e5-a278-d6f06e170348')#012 Jan 4 16:47:52 mds1 crmd: [1305]: info: process_lrm_event: LRM operation lustre-OST0000_3_monitor_0 (call=9, rc=0, cib-update=41, confirmed=true) ok Jan 4 16:47:52 mds1 crmd: [1305]: WARN: status_from_rc: Action 38 (lustre-OST0000_3_monitor_0) on mds1 failed (target: 7 vs. rc: 0): Error Jan 4 16:47:52 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0000_3_monitor_0, magic=0:0;38:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f, cib=0.248.6) : Event failed Jan 4 16:47:52 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Jan 4 16:47:52 mds1 crmd: [1305]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 16:47:52 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (38) confirmed on mds1 (rc=4) Jan 4 16:47:52 mds1 pengine: [1304]: info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/pengine/pe-input-40.bz2 Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_monitor_0 (13) confirmed on oss2 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_monitor_0 (14) confirmed on oss2 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_monitor_0 (15) confirmed on oss2 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_monitor_0 (16) confirmed on oss2 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_monitor_0 (4) confirmed on oss1 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_monitor_0 (5) confirmed on oss1 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_monitor_0 (7) confirmed on oss1 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_monitor_0 (6) confirmed on oss1 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_monitor_0 (22) confirmed on mds2 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_monitor_0 (23) confirmed on mds2 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_monitor_0 (24) confirmed on mds2 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action st-mds2_monitor_0 (25) confirmed on mds2 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: WARN: status_from_rc: Action 29 (lustre-OST0000_3_monitor_0) on mds2 failed (target: 7 vs. rc: 0): Error Jan 4 16:47:53 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0000_3_monitor_0, magic=0:0;29:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f, cib=0.248.19) : Event failed Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (29) confirmed on mds2 (rc=4) Jan 4 16:47:53 mds1 lrmd: [1302]: info: rsc:MGS_1:6: probe Jan 4 16:47:53 mds1 lrmd: [1302]: info: rsc:lustre-MDT0000_2:7: probe Jan 4 16:47:53 mds1 lrmd: [1302]: info: rsc:lustre-OST0001_4:8: probe Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (11) confirmed on oss1 (rc=0) Jan 4 16:47:53 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (10) confirmed on oss1 (rc=0) Jan 4 16:47:54 mds1 crmd: [1305]: info: process_lrm_event: LRM operation lustre-MDT0000_2_monitor_0 (call=7, rc=7, cib-update=42, confirmed=true) not running Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (36) confirmed on mds1 (rc=0) Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (20) confirmed on oss2 (rc=0) Jan 4 16:47:54 mds1 crmd: [1305]: WARN: status_from_rc: Action 8 (MGS_1_monitor_0) on oss1 failed (target: 7 vs. rc: 0): Error Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_monitor_0, magic=0:0;8:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f, cib=0.248.24) : Event failed Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_monitor_0 (8) confirmed on oss1 (rc=4) Jan 4 16:47:54 mds1 crmd: [1305]: WARN: status_from_rc: Action 9 (lustre-MDT0000_2_monitor_0) on oss1 failed (target: 7 vs. rc: 0): Error Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_monitor_0, magic=0:0;9:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f, cib=0.248.25) : Event failed Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (9) confirmed on oss1 (rc=4) Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 3: probe_complete probe_complete on oss1 - no waiting Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (19) confirmed on oss2 (rc=0) Jan 4 16:47:54 mds1 crmd: [1305]: info: process_lrm_event: LRM operation MGS_1_monitor_0 (call=6, rc=7, cib-update=43, confirmed=true) not running Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_monitor_0 (35) confirmed on mds1 (rc=0) Jan 4 16:47:54 mds1 lrmd: [1302]: info: RA output: (lustre-OST0001_4:probe:stderr) Traceback (most recent call last):#012#012 File "/usr/bin/hydra-agent.py", line 144, in #012 result = args.func(args)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py", line 315, in target_running#012 info = store_get_target_info(args.uuid)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/store.py", line 26, in store_get_target_info#012 raise RuntimeError("Failed to read target data for '%s', is it configured? (%s)" % (name, e))#012#012RuntimeError: Failed to read target data for '00f07899-ff44-4d74-b5e7-e42cde69d607', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/00f07899-ff44-4d74-b5e7-e42cde69d607')#012 Jan 4 16:47:54 mds1 crmd: [1305]: info: process_lrm_event: LRM operation lustre-OST0001_4_monitor_0 (call=8, rc=0, cib-update=44, confirmed=true) ok Jan 4 16:47:54 mds1 crmd: [1305]: WARN: status_from_rc: Action 37 (lustre-OST0001_4_monitor_0) on mds1 failed (target: 7 vs. rc: 0): Error Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0001_4_monitor_0, magic=0:0;37:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f, cib=0.248.28) : Event failed Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (37) confirmed on mds1 (rc=4) Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 30: probe_complete probe_complete on mds1 (local) - no waiting Jan 4 16:47:54 mds1 attrd: [1303]: info: find_hash_entry: Creating hash entry for probe_complete Jan 4 16:47:54 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Jan 4 16:47:54 mds1 attrd: [1303]: info: attrd_perform_update: Sent update 14: probe_complete=true Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.248.29) : Transient attribute: update Jan 4 16:47:54 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 1 to 1000000 Jan 4 16:47:54 mds1 crmd: [1305]: info: update_abort_priority: 'Event failed' abort superceeded Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.248.30) : Transient attribute: update Jan 4 16:47:54 mds1 crmd: [1305]: WARN: status_from_rc: Action 18 (lustre-MDT0000_2_monitor_0) on oss2 failed (target: 7 vs. rc: 0): Error Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_monitor_0, magic=0:0;18:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f, cib=0.248.31) : Event failed Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (18) confirmed on oss2 (rc=4) Jan 4 16:47:54 mds1 crmd: [1305]: WARN: status_from_rc: Action 17 (MGS_1_monitor_0) on oss2 failed (target: 7 vs. rc: 0): Error Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_monitor_0, magic=0:0;17:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f, cib=0.248.32) : Event failed Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_monitor_0 (17) confirmed on oss2 (rc=4) Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 12: probe_complete probe_complete on oss2 - no waiting Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss2, magic=NA, cib=0.248.33) : Transient attribute: update Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (27) confirmed on mds2 (rc=0) Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action MGS_1_monitor_0 (26) confirmed on mds2 (rc=0) Jan 4 16:47:54 mds1 crmd: [1305]: WARN: status_from_rc: Action 28 (lustre-OST0001_4_monitor_0) on mds2 failed (target: 7 vs. rc: 0): Error Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0001_4_monitor_0, magic=0:0;28:0:7:b4a0159a-ec35-4987-a4c9-c832425ca02f, cib=0.248.36) : Event failed Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (28) confirmed on mds2 (rc=4) Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 21: probe_complete probe_complete on mds2 - no waiting Jan 4 16:47:54 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 16:47:54 mds1 crmd: [1305]: notice: run_graph: Transition 0 (Complete=36, Pending=0, Fired=0, Skipped=13, Incomplete=0, Source=/var/lib/pengine/pe-input-40.bz2): Stopped Jan 4 16:47:54 mds1 crmd: [1305]: info: te_graph_trigger: Transition 0 is now complete Jan 4 16:47:54 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 16:47:54 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 16:47:54 mds1 crmd: [1305]: info: do_pe_invoke: Query 45: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:47:54 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=45, ref=pe_calc-dc-1325713674-54, seq=5052, quorate=1 Jan 4 16:47:54 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 16:47:54 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 16:47:54 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 16:47:54 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 16:47:54 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 16:47:54 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 16:47:54 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 16:47:54 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 16:47:54 mds1 pengine: [1304]: ERROR: native_add_running: Resource ocf::Target:lustre-MDT0000_2 appears to be active on 2 nodes. Jan 4 16:47:54 mds1 pengine: [1304]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:47:54 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 16:47:54 mds1 pengine: [1304]: ERROR: native_add_running: Resource ocf::Target:MGS_1 appears to be active on 2 nodes. Jan 4 16:47:54 mds1 pengine: [1304]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:47:54 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 16:47:54 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 16:47:54 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 16:47:54 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 16:47:54 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 16:47:54 mds1 pengine: [1304]: ERROR: native_add_running: Resource ocf::Target:lustre-OST0000_3 appears to be active on 2 nodes. Jan 4 16:47:54 mds1 pengine: [1304]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:47:54 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 16:47:54 mds1 pengine: [1304]: ERROR: native_add_running: Resource ocf::Target:lustre-OST0001_4 appears to be active on 2 nodes. Jan 4 16:47:54 mds1 pengine: [1304]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Stopped Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Stopped Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Stopped Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Stopped Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target) Started Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: #0110 : oss1 Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: #0111 : oss2 Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target) Started Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: #0110 : oss1 Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: #0111 : oss2 Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target) Started Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: #0110 : mds2 Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: #0111 : mds1 Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target) Started Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: #0110 : mds2 Jan 4 16:47:54 mds1 pengine: [1304]: notice: native_print: #0111 : mds1 Jan 4 16:47:54 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 16:47:54 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 16:47:54 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose mds2. Jan 4 16:47:54 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 16:47:54 mds1 pengine: [1304]: WARN: native_create_actions: Attempting recovery of resource MGS_1 Jan 4 16:47:54 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 16:47:54 mds1 pengine: [1304]: WARN: native_create_actions: Attempting recovery of resource lustre-MDT0000_2 Jan 4 16:47:54 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 16:47:54 mds1 pengine: [1304]: WARN: native_create_actions: Attempting recovery of resource lustre-OST0001_4 Jan 4 16:47:54 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 16:47:54 mds1 pengine: [1304]: WARN: native_create_actions: Attempting recovery of resource lustre-OST0000_3 Jan 4 16:47:54 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 16:47:54 mds1 pengine: [1304]: notice: LogActions: Start st-oss2#011(oss1) Jan 4 16:47:54 mds1 pengine: [1304]: notice: LogActions: Start st-oss1#011(oss2) Jan 4 16:47:54 mds1 pengine: [1304]: notice: LogActions: Start st-mds1#011(mds2) Jan 4 16:47:54 mds1 pengine: [1304]: notice: LogActions: Start st-mds2#011(mds1) Jan 4 16:47:54 mds1 pengine: [1304]: notice: LogActions: Move resource MGS_1#011(Started oss1 -> mds1) Jan 4 16:47:54 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started oss1 -> mds1) Jan 4 16:47:54 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0001_4#011(Started mds2 -> oss1) Jan 4 16:47:54 mds1 pengine: [1304]: notice: LogActions: Move resource lustre-OST0000_3#011(Started mds2 -> oss1) Jan 4 16:47:54 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 16:47:54 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 1: 22 actions in 22 synapses Jan 4 16:47:54 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1325713674-54) derived from /var/lib/pengine/pe-error-13.bz2 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 7: start st-oss2_start_0 on oss1 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 8: start st-oss1_start_0 on oss2 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 9: start st-mds1_start_0 on mds2 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 10: start st-mds2_start_0 on mds1 (local) Jan 4 16:47:54 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=10:1:0:b4a0159a-ec35-4987-a4c9-c832425ca02f op=st-mds2_start_0 ) Jan 4 16:47:54 mds1 lrmd: [1302]: info: rsc:st-mds2:10: start Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 11: stop MGS_1_stop_0 on oss1 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 12: stop MGS_1_stop_0 on oss2 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 15: stop lustre-MDT0000_2_stop_0 on oss1 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 16: stop lustre-MDT0000_2_stop_0 on oss2 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 19: stop lustre-OST0001_4_stop_0 on mds2 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 20: stop lustre-OST0001_4_stop_0 on mds1 (local) Jan 4 16:47:54 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=20:1:0:b4a0159a-ec35-4987-a4c9-c832425ca02f op=lustre-OST0001_4_stop_0 ) Jan 4 16:47:54 mds1 lrmd: [1302]: info: rsc:lustre-OST0001_4:11: stop Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 23: stop lustre-OST0000_3_stop_0 on mds2 Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 24: stop lustre-OST0000_3_stop_0 on mds1 (local) Jan 4 16:47:54 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=24:1:0:b4a0159a-ec35-4987-a4c9-c832425ca02f op=lustre-OST0000_3_stop_0 ) Jan 4 16:47:54 mds1 lrmd: [1302]: info: rsc:lustre-OST0000_3:12: stop Jan 4 16:47:54 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 5: probe_complete probe_complete on mds2 - no waiting Jan 4 16:47:54 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.248.37) : Transient attribute: update Jan 4 16:47:54 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Jan 4 16:47:54 mds1 crmd: [1305]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action st-oss2_start_0 (7) confirmed on oss1 (rc=0) Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action st-mds1_start_0 (9) confirmed on mds2 (rc=0) Jan 4 16:47:54 mds1 crmd: [1305]: info: match_graph_event: Action st-oss1_start_0 (8) confirmed on oss2 (rc=0) Jan 4 16:47:54 mds1 stonith-ng: [1300]: info: stonith_device_register: Added 'st-mds2' to the device list (1 active devices) Jan 4 16:47:54 mds1 stonith-ng: [1300]: info: stonith_command: Processed st_device_register from lrmd: rc=0 Jan 4 16:47:54 mds1 lrmd: [1302]: info: stonith_api_device_metadata: looking up fence_virsh/redhat metadata Jan 4 16:47:54 mds1 lrmd: [1302]: info: stonithRA plugin: got metadata: #012#012fence_virsh is an I/O Fencing agent which can be used with the virtual machines managed by libvirt. It logs via ssh to a dom0 and there run virsh command, which does all work. #012.P#012By default, virsh needs root account to do properly work. So you must allow ssh login in your sshd_config.#012#012#011#012#011#011#012#011#011#012#011#011Fencing Action#012#011#012#011#012#011#011#012#011#011#012#011#011IP Address or Hostname#012#011#012#011#012#011#011#012#011#011#012#011#011Login Name#012#011#012#011#012#011#011#012#011#011#012#011#011Login password or passphrase#012#011#012#011#012#011#011#012#011#011#012#011#011Script to retrieve password#012#011#012#011#012#011#011#012#011#011#012#011#011SSH connection#012#011#012#011#012#011#011#012 result = args.func(args)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py", line 230, in unmount_target#012 info = store_get_target_info(args.uuid)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/store.py", line 26, in store_get_target_info#012 raise RuntimeError("Failed to read target data for '%s', is it configured? (%s)" % (name, e))#012#012RuntimeError: Failed to read target data for '00f07899-ff44-4d74-b5e7-e42cde69d607', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/00f07899-ff44-4d74-b5e7-e42cde69d607')#012 Jan 4 16:47:55 mds1 lrmd: [1302]: info: RA output: (lustre-OST0000_3:stop:stderr) Traceback (most recent call last):#012#012 File "/usr/bin/hydra-agent.py", line 144, in #012 result = args.func(args)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py", line 230, in unmount_target#012 info = store_get_target_info(args.uuid)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/store.py", line 26, in store_get_target_info#012 raise RuntimeError("Failed to read target data for '%s', is it configured? (%s)" % (name, e))#012#012RuntimeError: Failed to read target data for '1a698033-3abc-41e5-a278-d6f06e170348', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/1a698033-3abc-41e5-a278-d6f06e170348')#012 Jan 4 16:47:55 mds1 lrmd: [1302]: info: RA output: (lustre-OST0001_4:stop:stdout) {#012 "backtrace": "Traceback (most recent call last):\n\n File \"/usr/bin/hydra-agent.py\", line 144, in \n result = args.func(args)\n\n File \"/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py\", line 230, in unmount_target\n info = store_get_target_info(args.uuid)\n\n File \"/usr/lib/python2.6/site-packages/hydra_agent/store.py\", line 26, in store_get_target_info\n raise RuntimeError(\"Failed to read target data for '%s', is it configured? (%s)\" % (name, e))\n\nRuntimeError: Failed to read target data for '00f07899-ff44-4d74-b5e7-e42cde69d607', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/00f07899-ff44-4d74-b5e7-e42cde69d607')\n", #012 "exception": "cexceptions\nRuntimeError\np0\n(S\"Failed to read target data for '00f07899-ff44-4d74-b5e7-e42cde69d607', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/00f07899-ff44-4d74-b5e7-e42cde69d607')\"\np1\ntp2\nRp3\n.", #012 "success": false#012} Jan 4 16:47:55 mds1 crmd: [1305]: info: process_lrm_event: LRM operation lustre-OST0001_4_stop_0 (call=11, rc=0, cib-update=47, confirmed=true) ok Jan 4 16:47:55 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_stop_0 (20) confirmed on mds1 (rc=0) Jan 4 16:47:55 mds1 lrmd: [1302]: info: RA output: (lustre-OST0000_3:stop:stdout) {#012 "backtrace": "Traceback (most recent call last):\n\n File \"/usr/bin/hydra-agent.py\", line 144, in \n result = args.func(args)\n\n File \"/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py\", line 230, in unmount_target\n info = store_get_target_info(args.uuid)\n\n File \"/usr/lib/python2.6/site-packages/hydra_agent/store.py\", line 26, in store_get_target_info\n raise RuntimeError(\"Failed to read target data for '%s', is it configured? (%s)\" % (name, e))\n\nRuntimeError: Failed to read target data for '1a698033-3abc-41e5-a278-d6f06e170348', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/1a698033-3abc-41e5-a278-d6f06e170348')\n", #012 "exception": "cexceptions\nRuntimeError\np0\n(S\"Failed to read target data for '1a698033-3abc-41e5-a278-d6f06e170348', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/1a698033-3abc-41e5-a278-d6f06e170348')\"\np1\ntp2\nRp3\n.", #012 "success": false#012} Jan 4 16:47:55 mds1 crmd: [1305]: info: process_lrm_event: LRM operation lustre-OST0000_3_stop_0 (call=12, rc=0, cib-update=48, confirmed=true) ok Jan 4 16:47:55 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_stop_0 (24) confirmed on mds1 (rc=0) Jan 4 16:47:55 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0001_4_stop_0 (19) confirmed on mds2 (rc=0) Jan 4 16:47:55 mds1 crmd: [1305]: info: match_graph_event: Action lustre-OST0000_3_stop_0 (23) confirmed on mds2 (rc=0) Jan 4 16:47:55 mds1 crmd: [1305]: info: run_graph: ==================================================== Jan 4 16:47:55 mds1 crmd: [1305]: notice: run_graph: Transition 1 (Complete=13, Pending=0, Fired=0, Skipped=9, Incomplete=0, Source=/var/lib/pengine/pe-error-13.bz2): Stopped Jan 4 16:47:55 mds1 crmd: [1305]: info: te_graph_trigger: Transition 1 is now complete Jan 4 16:47:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 16:47:55 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 16:47:55 mds1 crmd: [1305]: info: do_pe_invoke: Query 49: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:47:55 mds1 crmd: [1305]: info: do_pe_invoke_callback: Invoking the PE: query=49, ref=pe_calc-dc-1325713675-68, seq=5052, quorate=1 Jan 4 16:47:55 mds1 pengine: [1304]: info: unpack_config: Startup probes: enabled Jan 4 16:47:55 mds1 pengine: [1304]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 16:47:55 mds1 pengine: [1304]: info: unpack_domains: Unpacking domains Jan 4 16:47:55 mds1 pengine: [1304]: info: determine_online_status: Node oss1 is online Jan 4 16:47:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 16:47:55 mds1 pengine: [1304]: info: determine_online_status: Node oss2 is online Jan 4 16:47:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 16:47:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 16:47:55 mds1 pengine: [1304]: info: determine_online_status: Node mds2 is online Jan 4 16:47:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 16:47:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 16:47:55 mds1 pengine: [1304]: info: determine_online_status: Node mds1 is online Jan 4 16:47:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started oss1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started oss2 Jan 4 16:47:55 mds1 pengine: [1304]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds2 Jan 4 16:47:55 mds1 pengine: [1304]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 16:47:55 mds1 pengine: [1304]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 16:47:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 16:47:55 mds1 pengine: [1304]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 16:47:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 16:47:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 16:47:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose mds2. Jan 4 16:47:55 mds1 pengine: [1304]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 16:47:55 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 16:47:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss2#011(Started oss1) Jan 4 16:47:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-oss1#011(Started oss2) Jan 4 16:47:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds1#011(Started mds2) Jan 4 16:47:55 mds1 pengine: [1304]: notice: LogActions: Leave resource st-mds2#011(Started mds1) Jan 4 16:47:55 mds1 pengine: [1304]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 16:47:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 16:47:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0001_4#011(oss1) Jan 4 16:47:55 mds1 pengine: [1304]: notice: LogActions: Start lustre-OST0000_3#011(oss1) Jan 4 16:47:55 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 16:47:55 mds1 crmd: [1305]: info: unpack_graph: Unpacked transition 2: 8 actions in 8 synapses Jan 4 16:47:55 mds1 crmd: [1305]: info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1325713675-68) derived from /var/lib/pengine/pe-input-41.bz2 Jan 4 16:47:55 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 15: start MGS_1_start_0 on mds1 (local) Jan 4 16:47:55 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=15:2:0:b4a0159a-ec35-4987-a4c9-c832425ca02f op=MGS_1_start_0 ) Jan 4 16:47:55 mds1 lrmd: [1302]: info: rsc:MGS_1:13: start Jan 4 16:47:55 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 17: start lustre-MDT0000_2_start_0 on mds1 (local) Jan 4 16:47:55 mds1 crmd: [1305]: info: do_lrm_rsc_op: Performing key=17:2:0:b4a0159a-ec35-4987-a4c9-c832425ca02f op=lustre-MDT0000_2_start_0 ) Jan 4 16:47:55 mds1 lrmd: [1302]: info: rsc:lustre-MDT0000_2:14: start Jan 4 16:47:55 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 19: start lustre-OST0001_4_start_0 on oss1 Jan 4 16:47:55 mds1 crmd: [1305]: info: te_rsc_command: Initiating action 21: start lustre-OST0000_3_start_0 on oss1 Jan 4 16:47:55 mds1 pengine: [1304]: info: process_pe_message: Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-41.bz2 Jan 4 16:47:56 mds1 kernel: Lustre: OBD class driver, http://wiki.whamcloud.com/ Jan 4 16:47:56 mds1 kernel: Lustre: Lustre Version: 2.1.0 Jan 4 16:47:56 mds1 kernel: Lustre: Build Version: RC2-g9d71fe8-PRISTINE-2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 Jan 4 16:47:56 mds1 kernel: Lustre: Lustre LU module (ffffffffa052bf60). Jan 4 16:48:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:48:27 mds1 kernel: fld: Unknown symbol RQF_FLD_QUERY Jan 4 16:48:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:48:57 mds1 kernel: fld: Unknown symbol req_capsule_server_pack Jan 4 16:48:58 mds1 crmd: [1305]: info: handle_shutdown_request: Creating shutdown request for mds2 (state=S_TRANSITION_ENGINE) Jan 4 16:48:58 mds1 corosync[1290]: [SERV ] Unloading all Corosync service engines. Jan 4 16:48:58 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Shuting down Pacemaker Jan 4 16:48:58 mds1 corosync[1290]: [pcmk ] notice: stop_child: Sent -15 to crmd: [1305] Jan 4 16:48:58 mds1 crmd: [1305]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 16:48:58 mds1 crmd: [1305]: info: crm_shutdown: Requesting shutdown Jan 4 16:48:58 mds1 crmd: [1305]: WARN: do_log: FSA: Input I_SHUTDOWN from crm_shutdown() received in state S_TRANSITION_ENGINE Jan 4 16:48:58 mds1 crmd: [1305]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ] Jan 4 16:48:58 mds1 crmd: [1305]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 16:48:58 mds1 crmd: [1305]: info: do_shutdown_req: Sending shutdown request to DC: mds1 Jan 4 16:48:58 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.248.51) : Transient attribute: update Jan 4 16:48:58 mds1 crmd: [1305]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Jan 4 16:48:58 mds1 crmd: [1305]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 16:48:58 mds1 crmd: [1305]: info: handle_shutdown_request: Creating shutdown request for mds1 (state=S_POLICY_ENGINE) Jan 4 16:48:58 mds1 crmd: [1305]: info: handle_shutdown_request: Creating shutdown request for oss1 (state=S_POLICY_ENGINE) Jan 4 16:48:58 mds1 attrd: [1303]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (1325713738) Jan 4 16:48:58 mds1 attrd: [1303]: info: attrd_perform_update: Sent update 24: shutdown=1325713738 Jan 4 16:48:58 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.248.54) : Transient attribute: update Jan 4 16:48:58 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.248.55) : Transient attribute: update Jan 4 16:48:58 mds1 crmd: [1305]: info: handle_shutdown_request: Creating shutdown request for oss2 (state=S_POLICY_ENGINE) Jan 4 16:48:58 mds1 crmd: [1305]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss2, magic=NA, cib=0.248.57) : Transient attribute: update Jan 4 16:49:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:49:27 mds1 kernel: fld: Unknown symbol req_capsule_client_get Jan 4 16:49:28 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1305, seq=6) to terminate... Jan 4 16:49:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:49:57 mds1 kernel: fld: Unknown symbol ptlrpc_queue_wait Jan 4 16:49:58 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1305, seq=6) to terminate... Jan 4 16:50:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:50:27 mds1 kernel: fld: Unknown symbol req_capsule_fini Jan 4 16:50:28 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1305, seq=6) to terminate... Jan 4 16:50:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:50:57 mds1 kernel: fld: Unknown symbol req_capsule_init Jan 4 16:50:58 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1305, seq=6) to terminate... Jan 4 16:51:04 mds1 stonith-ng: [1300]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 16:51:04 mds1 stonith-ng: [1300]: info: stonith_shutdown: Terminating with 2 clients Jan 4 16:51:04 mds1 crmd: [1305]: info: stonith_msgready: Lost connection to the STONITH service [1300]. Jan 4 16:51:04 mds1 crmd: [1305]: CRIT: stonith_dispatch: Lost connection to the STONITH service [1300/callback]. Jan 4 16:51:04 mds1 crmd: [1305]: CRIT: stonith_dispatch: Lost connection to the STONITH service [1300/command]. Jan 4 16:51:04 mds1 crmd: [1305]: CRIT: tengine_stonith_connection_destroy: Fencing daemon connection failed Jan 4 16:51:04 mds1 crmd: [1305]: info: te_connect_stonith: Attempting connection to fencing daemon... Jan 4 16:51:04 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc_exit: Client stonith-ng (conn=0x242b4c0, async-conn=0x242b4c0) left Jan 4 16:51:04 mds1 cib: [1301]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 16:51:04 mds1 cib: [1301]: WARN: disconnect_cib_client: Disconnecting 1305/cib_callback... Jan 4 16:51:04 mds1 cib: [1301]: WARN: disconnect_cib_client: Disconnecting 1303/cib_callback... Jan 4 16:51:04 mds1 cib: [1301]: WARN: disconnect_cib_client: Disconnecting crmd/cib_rw... Jan 4 16:51:04 mds1 cib: [1301]: WARN: disconnect_cib_client: Disconnecting attrd/cib_rw... Jan 4 16:51:04 mds1 cib: [1301]: info: cib_shutdown: Disconnected 4 clients Jan 4 16:51:04 mds1 cib: [1301]: info: cib_process_disconnect: All clients disconnected... Jan 4 16:51:04 mds1 cib: [1301]: info: initiate_exit: Sending disconnect notification to 4 peers... Jan 4 16:51:04 mds1 attrd: [1303]: info: cib_native_msgready: Lost connection to the CIB service [1301]. Jan 4 16:51:04 mds1 attrd: [1303]: CRIT: cib_native_dispatch: Lost connection to the CIB service [1301/callback]. Jan 4 16:51:04 mds1 attrd: [1303]: CRIT: cib_native_dispatch: Lost connection to the CIB service [1301/command]. Jan 4 16:51:04 mds1 attrd: [1303]: ERROR: attrd_cib_connection_destroy: Connection to the CIB terminated... Jan 4 16:51:04 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc_exit: Client attrd (conn=0x242fc90, async-conn=0x242fc90) left Jan 4 16:51:04 mds1 lrmd: [1302]: info: sigterm_action: shutdown postponed, some operations are still running Jan 4 16:51:04 mds1 cib: [1301]: info: cib_process_shutdown_req: Shutdown ACK from oss2 Jan 4 16:51:04 mds1 cib: [1301]: info: cib_ha_connection_destroy: Heartbeat disconnection complete... exiting Jan 4 16:51:04 mds1 cib: [1301]: info: cib_ha_connection_destroy: Exiting... Jan 4 16:51:04 mds1 cib: [1301]: info: cib_process_request: Operation complete: op cib_shutdown_req for section 'all' (origin=oss2/oss2/(null), version=0.0.0): ok (rc=0) Jan 4 16:51:04 mds1 cib: [1301]: info: main: Done Jan 4 16:51:04 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc_exit: Client cib (conn=0x2433ff0, async-conn=0x2433ff0) left Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: te_connect_stonith: Sign-in failed: triggered a retry Jan 4 16:51:05 mds1 crmd: [1305]: info: cib_native_msgready: Lost connection to the CIB service [1301]. Jan 4 16:51:05 mds1 crmd: [1305]: CRIT: cib_native_dispatch: Lost connection to the CIB service [1301/callback]. Jan 4 16:51:05 mds1 crmd: [1305]: CRIT: cib_native_dispatch: Lost connection to the CIB service [1301/command]. Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: crmd_cib_connection_destroy: Connection to the CIB terminated... Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: do_log: FSA: Input I_ERROR from crmd_cib_connection_destroy() received in state S_POLICY_ENGINE Jan 4 16:51:05 mds1 crmd: [1305]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_RECOVERY [ input=I_ERROR cause=C_FSA_INTERNAL origin=crmd_cib_connection_destroy ] Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: do_recover: Action A_RECOVER (0000000001000000) not supported Jan 4 16:51:05 mds1 crmd: [1305]: WARN: do_election_vote: Not voting in election, we're in state S_RECOVERY Jan 4 16:51:05 mds1 crmd: [1305]: info: do_dc_release: DC role released Jan 4 16:51:05 mds1 crmd: [1305]: info: pe_connection_destroy: Connection to the Policy Engine released Jan 4 16:51:05 mds1 crmd: [1305]: WARN: destroy_action: Cancelling timer for action 15 (src=97) Jan 4 16:51:05 mds1 crmd: [1305]: WARN: destroy_action: Cancelling timer for action 17 (src=98) Jan 4 16:51:05 mds1 crmd: [1305]: WARN: destroy_action: Cancelling timer for action 19 (src=99) Jan 4 16:51:05 mds1 crmd: [1305]: WARN: destroy_action: Cancelling timer for action 21 (src=100) Jan 4 16:51:05 mds1 crmd: [1305]: info: do_te_control: Transitioner is now inactive Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: do_log: FSA: Input I_TERMINATE from do_recover() received in state S_RECOVERY Jan 4 16:51:05 mds1 crmd: [1305]: info: do_state_transition: State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_INTERNAL origin=do_recover ] Jan 4 16:51:05 mds1 crmd: [1305]: info: do_shutdown: Disconnecting STONITH... Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: verify_stopped: 2 pending LRM operations at shutdown Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: ghash_print_pending: Pending action: MGS_1:13 (MGS_1_start_0) Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: ghash_print_pending: Pending action: lustre-MDT0000_2:14 (lustre-MDT0000_2_start_0) Jan 4 16:51:05 mds1 crmd: [1305]: info: do_lrm_control: Disconnected from the LRM Jan 4 16:51:05 mds1 crmd: [1305]: info: do_ha_control: Disconnected from OpenAIS Jan 4 16:51:05 mds1 crmd: [1305]: info: do_cib_control: Disconnecting CIB Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: verify_stopped: 2 pending LRM operations at shutdown Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: ghash_print_pending: Pending action: MGS_1:13 (MGS_1_start_0) Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: ghash_print_pending: Pending action: lustre-MDT0000_2:14 (lustre-MDT0000_2_start_0) Jan 4 16:51:05 mds1 crmd: [1305]: info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd Jan 4 16:51:05 mds1 crmd: [1305]: ERROR: do_exit: Could not recover from internal error Jan 4 16:51:05 mds1 crmd: [1305]: info: free_mem: Dropping I_PENDING: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_election_vote ] Jan 4 16:51:05 mds1 crmd: [1305]: info: free_mem: Dropping I_RELEASE_SUCCESS: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_dc_release ] Jan 4 16:51:05 mds1 crmd: [1305]: info: free_mem: Dropping I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_stop ] Jan 4 16:51:05 mds1 crmd: [1305]: info: do_exit: [crmd] stopped (2) Jan 4 16:51:05 mds1 corosync[1290]: [pcmk ] info: pcmk_ipc_exit: Client crmd (conn=0x2438350, async-conn=0x2438350) left Jan 4 16:51:05 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: crmd confirmed stopped Jan 4 16:51:05 mds1 corosync[1290]: [pcmk ] notice: stop_child: Sent -15 to pengine: [1304] Jan 4 16:51:05 mds1 pengine: [1304]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 16:51:06 mds1 lrmd: [1302]: info: sigterm_action: shutdown postponed, some operations are still running Jan 4 16:51:06 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: pengine confirmed stopped Jan 4 16:51:06 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: attrd confirmed stopped Jan 4 16:51:06 mds1 corosync[1290]: [pcmk ] notice: stop_child: Sent -15 to lrmd: [1302] Jan 4 16:51:06 mds1 lrmd: [1302]: info: sigterm_action: shutdown postponed, some operations are still running Jan 4 16:51:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:51:27 mds1 kernel: fld: Unknown symbol req_capsule_set Jan 4 16:51:27 mds1 init: tty (/dev/tty1) main process (1576) killed by TERM signal Jan 4 16:51:27 mds1 init: tty (/dev/tty2) main process (1578) killed by TERM signal Jan 4 16:51:27 mds1 init: tty (/dev/tty3) main process (1580) killed by TERM signal Jan 4 16:51:27 mds1 init: tty (/dev/tty4) main process (1582) killed by TERM signal Jan 4 16:51:27 mds1 init: tty (/dev/tty5) main process (1584) killed by TERM signal Jan 4 16:51:27 mds1 init: tty (/dev/tty6) main process (1586) killed by TERM signal Jan 4 16:51:27 mds1 avahi-dnsconfd[1557]: Got SIGTERM, quitting. Jan 4 16:51:27 mds1 avahi-daemon[1330]: Got SIGTERM, quitting. Jan 4 16:51:27 mds1 avahi-daemon[1330]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.122.89. Jan 4 16:51:28 mds1 acpid: exiting Jan 4 16:51:36 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Still waiting for lrmd (pid=1302, seq=3) to terminate... Jan 4 16:51:55 mds1 kernel: INFO: task modprobe:1651 blocked for more than 120 seconds. Jan 4 16:51:55 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 4 16:51:55 mds1 kernel: modprobe D 0000000000000000 0 1651 1643 0x00000080 Jan 4 16:51:55 mds1 kernel: ffff88003d7b5aa8 0000000000000082 0000000000000000 ffff88003e7260f8 Jan 4 16:51:55 mds1 kernel: ffff880002015f80 ffff88003e7260c0 ffff880002015f80 0000000000000000 Jan 4 16:51:55 mds1 kernel: ffff88003d0da5f8 ffff88003d7b5fd8 000000000000f598 ffff88003d0da5f8 Jan 4 16:51:55 mds1 kernel: Call Trace: Jan 4 16:51:55 mds1 kernel: [] schedule_timeout+0x215/0x2e0 Jan 4 16:51:55 mds1 kernel: [] ? autoremove_wake_function+0x16/0x40 Jan 4 16:51:55 mds1 kernel: [] ? __wake_up_common+0x59/0x90 Jan 4 16:51:55 mds1 kernel: [] wait_for_common+0x123/0x180 Jan 4 16:51:55 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 16:51:55 mds1 kernel: [] ? __queue_work+0x41/0x50 Jan 4 16:51:55 mds1 kernel: [] wait_for_completion+0x1d/0x20 Jan 4 16:51:55 mds1 kernel: [] call_usermodehelper_exec+0xe0/0xf0 Jan 4 16:51:55 mds1 kernel: [] ? lnet_startup_lndnis+0xe6/0x6e0 [lnet] Jan 4 16:51:55 mds1 kernel: [] __request_module+0x18b/0x210 Jan 4 16:51:55 mds1 kernel: [] ? lnet_parse_networks+0xb0/0x7e0 [lnet] Jan 4 16:51:55 mds1 kernel: [] ? pvclock_clocksource_read+0x58/0xd0 Jan 4 16:51:55 mds1 kernel: [] lnet_startup_lndnis+0xe6/0x6e0 [lnet] Jan 4 16:51:55 mds1 kernel: [] ? cfs_alloc+0x63/0x90 [libcfs] Jan 4 16:51:55 mds1 kernel: [] LNetNIInit+0x125/0x1f0 [lnet] Jan 4 16:51:55 mds1 kernel: [] ? __cond_resched+0x2a/0x40 Jan 4 16:51:55 mds1 kernel: [] ? init_module+0x0/0x597 [ptlrpc] Jan 4 16:51:55 mds1 kernel: [] ptlrpc_ni_init+0x29/0x170 [ptlrpc] Jan 4 16:51:55 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 16:51:55 mds1 kernel: [] ptlrpc_init_portals+0x13/0xd0 [ptlrpc] Jan 4 16:51:55 mds1 kernel: [] ? init_module+0x0/0x597 [ptlrpc] Jan 4 16:51:55 mds1 kernel: [] init_module+0xe0/0x597 [ptlrpc] Jan 4 16:51:55 mds1 kernel: [] ? __blocking_notifier_call_chain+0x65/0x80 Jan 4 16:51:55 mds1 kernel: [] do_one_initcall+0x3c/0x1d0 Jan 4 16:51:55 mds1 kernel: [] sys_init_module+0xdf/0x250 Jan 4 16:51:55 mds1 kernel: [] system_call_fastpath+0x16/0x1b Jan 4 16:51:55 mds1 kernel: INFO: task mount.lustre:1696 blocked for more than 120 seconds. Jan 4 16:51:55 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 4 16:51:55 mds1 kernel: mount.lustre D 0000000000000000 0 1696 1 0x00000084 Jan 4 16:51:55 mds1 kernel: ffff88002eebd908 0000000000000086 ffff88002eebd8b8 ffffffff810097bc Jan 4 16:51:55 mds1 kernel: 0000000000000000 ffff88003e7260c0 00000000007260f8 ffff880002013b40 Jan 4 16:51:55 mds1 kernel: ffff88002ecb7b38 ffff88002eebdfd8 000000000000f598 ffff88002ecb7b38 Jan 4 16:51:55 mds1 kernel: Call Trace: Jan 4 16:51:55 mds1 kernel: [] ? __switch_to+0x1ac/0x320 Jan 4 16:51:55 mds1 kernel: [] ? thread_return+0x652/0x777 Jan 4 16:51:55 mds1 kernel: [] schedule_timeout+0x215/0x2e0 Jan 4 16:51:55 mds1 kernel: [] ? __wake_up_common+0x59/0x90 Jan 4 16:51:55 mds1 kernel: [] wait_for_common+0x123/0x180 Jan 4 16:51:55 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 16:51:55 mds1 kernel: [] ? __queue_work+0x41/0x50 Jan 4 16:51:55 mds1 kernel: [] wait_for_completion+0x1d/0x20 Jan 4 16:51:55 mds1 kernel: [] call_usermodehelper_exec+0xe0/0xf0 Jan 4 16:51:55 mds1 kernel: [] ? get_fs_type+0xa1/0xd0 Jan 4 16:51:55 mds1 kernel: [] __request_module+0x18b/0x210 Jan 4 16:51:55 mds1 kernel: [] ? find_filesystem+0x50/0x80 Jan 4 16:51:55 mds1 kernel: [] get_fs_type+0xa1/0xd0 Jan 4 16:51:55 mds1 kernel: [] server_kernel_mount+0x1ad/0xe90 [obdclass] Jan 4 16:51:55 mds1 kernel: [] ? lustre_fill_super+0x0/0x1280 [obdclass] Jan 4 16:51:55 mds1 kernel: [] ? lustre_init_lsi+0xd0/0x5c0 [obdclass] Jan 4 16:51:55 mds1 kernel: [] lustre_fill_super+0x278/0x1280 [obdclass] Jan 4 16:51:55 mds1 kernel: [] ? sget+0x3f9/0x490 Jan 4 16:51:55 mds1 kernel: [] ? set_anon_super+0x0/0x100 Jan 4 16:51:55 mds1 kernel: [] ? lustre_fill_super+0x0/0x1280 [obdclass] Jan 4 16:51:55 mds1 kernel: [] get_sb_nodev+0x5f/0xa0 Jan 4 16:51:55 mds1 kernel: [] lustre_get_sb+0x25/0x30 [obdclass] Jan 4 16:51:55 mds1 kernel: [] vfs_kern_mount+0x7b/0x1b0 Jan 4 16:51:55 mds1 kernel: [] do_kern_mount+0x52/0x130 Jan 4 16:51:55 mds1 kernel: [] do_mount+0x2e7/0x870 Jan 4 16:51:55 mds1 kernel: [] sys_mount+0x90/0xe0 Jan 4 16:51:55 mds1 kernel: [] system_call_fastpath+0x16/0x1b Jan 4 16:51:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:51:57 mds1 kernel: fld: Unknown symbol req_capsule_server_get Jan 4 16:52:06 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Still waiting for lrmd (pid=1302, seq=3) to terminate... Jan 4 16:52:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:52:27 mds1 kernel: fld: Unknown symbol ptlrpc_at_set_req_timeout Jan 4 16:52:36 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Still waiting for lrmd (pid=1302, seq=3) to terminate... Jan 4 16:52:55 mds1 lrmd: [1302]: WARN: MGS_1:start process (PID 1642) timed out (try 1). Killing with signal SIGTERM (15). Jan 4 16:52:55 mds1 lrmd: [1302]: WARN: lustre-MDT0000_2:start process (PID 1643) timed out (try 1). Killing with signal SIGTERM (15). Jan 4 16:52:55 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:52:55 mds1 kernel: fld: Unknown symbol ptlrpc_request_alloc_pack Jan 4 16:52:55 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:52:55 mds1 kernel: fld: Unknown symbol RMF_FLD_OPC Jan 4 16:52:55 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:52:55 mds1 kernel: fld: Unknown symbol ptlrpc_request_set_replen Jan 4 16:52:55 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:52:55 mds1 kernel: fld: Unknown symbol RMF_FLD_MDFLD Jan 4 16:52:55 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:52:55 mds1 kernel: fld: Unknown symbol ptlrpc_req_finished Jan 4 16:52:55 mds1 lrmd: [1302]: WARN: operation start[13] on ocf::Target::MGS_1 for client 1305, its parameters: CRM_meta_name=[start] crm_feature_set=[3.0.2] target=[96c8dccb-368b-4ffc-af7e-6b95cdcacadb] CRM_meta_timeout=[300000] : pid [1642] timed out Jan 4 16:52:55 mds1 lrmd: [1302]: info: record_op_completion: cannot record operation start[13] on ocf::Target::MGS_1 for client 1305: the client is gone Jan 4 16:52:55 mds1 lrmd: [1302]: WARN: notify_client: client for the operation operation start[13] on ocf::Target::MGS_1 for client 1305, its parameters: CRM_meta_name=[start] crm_feature_set=[3.0.2] target=[96c8dccb-368b-4ffc-af7e-6b95cdcacadb] CRM_meta_timeout=[300000] does not exist and client requested notification. Jan 4 16:52:55 mds1 lrmd: [1302]: WARN: operation start[14] on ocf::Target::lustre-MDT0000_2 for client 1305, its parameters: CRM_meta_name=[start] crm_feature_set=[3.0.2] target=[c4f08837-0cea-46c1-a5d8-4ef412fadd0b] CRM_meta_timeout=[300000] : pid [1643] timed out Jan 4 16:52:55 mds1 lrmd: [1302]: info: record_op_completion: cannot record operation start[14] on ocf::Target::lustre-MDT0000_2 for client 1305: the client is gone Jan 4 16:52:55 mds1 lrmd: [1302]: WARN: notify_client: client for the operation operation start[14] on ocf::Target::lustre-MDT0000_2 for client 1305, its parameters: CRM_meta_name=[start] crm_feature_set=[3.0.2] target=[c4f08837-0cea-46c1-a5d8-4ef412fadd0b] CRM_meta_timeout=[300000] does not exist and client requested notification. Jan 4 16:52:55 mds1 lrmd: [1302]: info: lrmd is shutting down Jan 4 16:52:55 mds1 lrmd: [1302]: info: lrmd is shutting down Jan 4 16:52:55 mds1 corosync[1290]: [pcmk ] WARN: route_ais_message: Sending message to local.cib failed: ipc delivery failed (rc=-2) Jan 4 16:52:55 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: lrmd confirmed stopped Jan 4 16:52:55 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: cib confirmed stopped Jan 4 16:52:55 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: stonith-ng confirmed stopped Jan 4 16:52:55 mds1 corosync[1290]: [pcmk ] info: update_member: Node mds1 now has process list: 00000000000000000000000000000002 (2) Jan 4 16:52:55 mds1 corosync[1290]: [pcmk ] notice: pcmk_shutdown: Shutdown complete Jan 4 16:52:55 mds1 corosync[1290]: [SERV ] Service engine unloaded: Pacemaker Cluster Manager 1.1.2 Jan 4 16:52:55 mds1 kernel: LustreError: 1651:0:(socklnd.c:2420:ksocknal_base_startup()) Can't spawn socknal scheduler[0]: -513 Jan 4 16:52:55 mds1 kernel: LustreError: 105-4: Error -100 starting up LNI tcp Jan 4 16:52:55 mds1 kernel: LustreError: 1651:0:(events.c:728:ptlrpc_init_portals()) network initialisation failed Jan 4 16:52:55 mds1 corosync[1290]: [SERV ] Service engine unloaded: corosync extended virtual synchrony service Jan 4 16:52:55 mds1 corosync[1290]: [SERV ] Service engine unloaded: corosync configuration service Jan 4 16:52:55 mds1 corosync[1290]: [SERV ] Service engine unloaded: corosync cluster closed process group service v1.01 Jan 4 16:52:55 mds1 corosync[1290]: [SERV ] Service engine unloaded: corosync cluster config database access v1.01 Jan 4 16:52:55 mds1 corosync[1290]: [SERV ] Service engine unloaded: corosync profile loading service Jan 4 16:52:55 mds1 corosync[1290]: [SERV ] Service engine unloaded: corosync cluster quorum service v0.1 Jan 4 16:52:55 mds1 corosync[1290]: [MAIN ] Corosync Cluster Engine exiting with status 0 at main.c:170. Jan 4 16:52:55 mds1 kernel: init dynlocks cache Jan 4 16:52:55 mds1 kernel: LDISKFS-fs warning (device sdb): ldiskfs_fill_super: extents feature not enabled on this filesystem, use tune2fs. Jan 4 16:52:55 mds1 kernel: LDISKFS-fs (sdb): mounted filesystem with ordered data mode Jan 4 16:52:55 mds1 kernel: LustreError: 1696:0:(obd_mount.c:309:ldd_parse()) error reading CONFIGS/mountdata: read -4 of 12288 Jan 4 16:52:55 mds1 kernel: LustreError: 1696:0:(obd_mount.c:1373:server_kernel_mount()) premount parse options failed: rc = -22 Jan 4 16:52:55 mds1 kernel: LustreError: 1696:0:(obd_mount.c:1681:server_fill_super()) Unable to mount device /dev/sdb: -22 Jan 4 16:52:55 mds1 kernel: LustreError: 1696:0:(obd_mount.c:2160:lustre_fill_super()) Unable to mount (-22) Jan 4 16:53:19 mds1 kernel: imklog 4.6.2, log source = /proc/kmsg started. Jan 4 16:53:19 mds1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1191" x-info="http://www.rsyslog.com"] (re)start Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys cpuset Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys cpu Jan 4 16:53:19 mds1 kernel: Linux version 2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 (jenkins@rhel6-64-build01.lab.whamcloud.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Sat Aug 20 09:56:49 PDT 2011 Jan 4 16:53:19 mds1 kernel: Command line: ro root=/dev/mapper/LustreVG-root rd_LVM_LV=LustreVG/root rd_LVM_LV=LustreVG/swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet Jan 4 16:53:19 mds1 kernel: KERNEL supported cpus: Jan 4 16:53:19 mds1 kernel: Intel GenuineIntel Jan 4 16:53:19 mds1 kernel: AMD AuthenticAMD Jan 4 16:53:19 mds1 kernel: Centaur CentaurHauls Jan 4 16:53:19 mds1 kernel: BIOS-provided physical RAM map: Jan 4 16:53:19 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) Jan 4 16:53:19 mds1 kernel: BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) Jan 4 16:53:19 mds1 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) Jan 4 16:53:19 mds1 kernel: BIOS-e820: 0000000000100000 - 000000003fffd000 (usable) Jan 4 16:53:19 mds1 kernel: BIOS-e820: 000000003fffd000 - 0000000040000000 (reserved) Jan 4 16:53:19 mds1 kernel: BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved) Jan 4 16:53:19 mds1 kernel: BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved) Jan 4 16:53:19 mds1 kernel: DMI 2.4 present. Jan 4 16:53:19 mds1 kernel: SMBIOS version 2.4 @ 0xFD760 Jan 4 16:53:19 mds1 kernel: last_pfn = 0x3fffd max_arch_pfn = 0x400000000 Jan 4 16:53:19 mds1 kernel: PAT not supported by CPU. Jan 4 16:53:19 mds1 kernel: init_memory_mapping: 0000000000000000-000000003fffd000 Jan 4 16:53:19 mds1 kernel: RAMDISK: 34a6c000 - 37fef97b Jan 4 16:53:19 mds1 kernel: ACPI: RSDP 00000000000fd730 00014 (v00 BOCHS ) Jan 4 16:53:19 mds1 kernel: ACPI: RSDT 000000003fffdc40 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) Jan 4 16:53:19 mds1 kernel: ACPI: FACP 000000003ffffe70 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) Jan 4 16:53:19 mds1 kernel: ACPI: DSDT 000000003fffde40 01FB7 (v01 BXPC BXDSDT 00000001 INTL 20090123) Jan 4 16:53:19 mds1 kernel: ACPI: FACS 000000003ffffe00 00040 Jan 4 16:53:19 mds1 kernel: ACPI: SSDT 000000003fffdda0 0009E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) Jan 4 16:53:19 mds1 kernel: ACPI: APIC 000000003fffdcc0 00072 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) Jan 4 16:53:19 mds1 kernel: ACPI: HPET 000000003fffdc80 00038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001) Jan 4 16:53:19 mds1 kernel: No NUMA configuration found Jan 4 16:53:19 mds1 kernel: Faking a node at 0000000000000000-000000003fffd000 Jan 4 16:53:19 mds1 kernel: Bootmem setup node 0 0000000000000000-000000003fffd000 Jan 4 16:53:19 mds1 kernel: NODE_DATA [0000000000009000 - 000000000003cfff] Jan 4 16:53:19 mds1 kernel: bootmap [000000000003d000 - 0000000000044fff] pages 8 Jan 4 16:53:19 mds1 kernel: (7 early reservations) ==> bootmem [0000000000 - 003fffd000] Jan 4 16:53:19 mds1 kernel: #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] Jan 4 16:53:19 mds1 kernel: #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] Jan 4 16:53:19 mds1 kernel: #2 [0001000000 - 0001f474e4] TEXT DATA BSS ==> [0001000000 - 0001f474e4] Jan 4 16:53:19 mds1 kernel: #3 [0034a6c000 - 0037fef97b] RAMDISK ==> [0034a6c000 - 0037fef97b] Jan 4 16:53:19 mds1 kernel: #4 [000009dc00 - 0000100000] BIOS reserved ==> [000009dc00 - 0000100000] Jan 4 16:53:19 mds1 kernel: #5 [0001f48000 - 0001f48079] BRK ==> [0001f48000 - 0001f48079] Jan 4 16:53:19 mds1 kernel: #6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000] Jan 4 16:53:19 mds1 kernel: found SMP MP-table at [ffff8800000fd780] fd780 Jan 4 16:53:19 mds1 kernel: kvm-clock: Using msrs 12 and 11 Jan 4 16:53:19 mds1 kernel: kvm-clock: cpu 0, msr 0:1bbbf01, boot clock Jan 4 16:53:19 mds1 kernel: Zone PFN ranges: Jan 4 16:53:19 mds1 kernel: DMA 0x00000001 -> 0x00001000 Jan 4 16:53:19 mds1 kernel: DMA32 0x00001000 -> 0x00100000 Jan 4 16:53:19 mds1 kernel: Normal 0x00100000 -> 0x00100000 Jan 4 16:53:19 mds1 kernel: Movable zone start PFN for each node Jan 4 16:53:19 mds1 kernel: early_node_map[2] active PFN ranges Jan 4 16:53:19 mds1 kernel: 0: 0x00000001 -> 0x0000009d Jan 4 16:53:19 mds1 kernel: 0: 0x00000100 -> 0x0003fffd Jan 4 16:53:19 mds1 kernel: ACPI: PM-Timer IO Port: 0xb008 Jan 4 16:53:19 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Jan 4 16:53:19 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) Jan 4 16:53:19 mds1 kernel: IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23 Jan 4 16:53:19 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Jan 4 16:53:19 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) Jan 4 16:53:19 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Jan 4 16:53:19 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) Jan 4 16:53:19 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) Jan 4 16:53:19 mds1 kernel: Using ACPI (MADT) for SMP configuration information Jan 4 16:53:19 mds1 kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000 Jan 4 16:53:19 mds1 kernel: SMP: Allowing 1 CPUs, 0 hotplug CPUs Jan 4 16:53:19 mds1 kernel: PM: Registered nosave memory: 000000000009d000 - 000000000009e000 Jan 4 16:53:19 mds1 kernel: PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 Jan 4 16:53:19 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 Jan 4 16:53:19 mds1 kernel: PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 Jan 4 16:53:19 mds1 kernel: Allocating PCI resources starting at 40000000 (gap: 40000000:beffc000) Jan 4 16:53:19 mds1 kernel: Booting paravirtualized kernel on KVM Jan 4 16:53:19 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 Jan 4 16:53:19 mds1 kernel: PERCPU: Embedded 30 pages/cpu @ffff880002000000 s92504 r8192 d22184 u2097152 Jan 4 16:53:19 mds1 kernel: pcpu-alloc: s92504 r8192 d22184 u2097152 alloc=1*2097152 Jan 4 16:53:19 mds1 kernel: pcpu-alloc: [0] 0 Jan 4 16:53:19 mds1 kernel: kvm-clock: cpu 0, msr 0:2015f01, primary cpu clock Jan 4 16:53:19 mds1 kernel: Built 1 zonelists in Node order, mobility grouping on. Total pages: 258355 Jan 4 16:53:19 mds1 kernel: Policy zone: DMA32 Jan 4 16:53:19 mds1 kernel: Kernel command line: ro root=/dev/mapper/LustreVG-root rd_LVM_LV=LustreVG/root rd_LVM_LV=LustreVG/swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet Jan 4 16:53:19 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes) Jan 4 16:53:19 mds1 kernel: Checking aperture... Jan 4 16:53:19 mds1 kernel: No AGP bridge found Jan 4 16:53:19 mds1 kernel: AMD-Vi disabled by default: pass amd_iommu=on to enable Jan 4 16:53:19 mds1 kernel: Memory: 962756k/1048564k available (5016k kernel code, 400k absent, 85408k reserved, 6904k data, 1232k init) Jan 4 16:53:19 mds1 kernel: Hierarchical RCU implementation. Jan 4 16:53:19 mds1 kernel: NR_IRQS:33024 nr_irqs:256 Jan 4 16:53:19 mds1 kernel: Console: colour VGA+ 80x25 Jan 4 16:53:19 mds1 kernel: console [tty0] enabled Jan 4 16:53:19 mds1 kernel: allocated 10485760 bytes of page_cgroup Jan 4 16:53:19 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Jan 4 16:53:19 mds1 kernel: HPET: 3 timers in total, 0 timers will be used for per-cpu timer Jan 4 16:53:19 mds1 kernel: Detected 1595.883 MHz processor. Jan 4 16:53:19 mds1 kernel: Calibrating delay loop (skipped) preset value.. 3191.76 BogoMIPS (lpj=1595883) Jan 4 16:53:19 mds1 kernel: pid_max: default: 32768 minimum: 301 Jan 4 16:53:19 mds1 kernel: Security Framework initialized Jan 4 16:53:19 mds1 kernel: SELinux: Initializing. Jan 4 16:53:19 mds1 kernel: Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Jan 4 16:53:19 mds1 kernel: Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) Jan 4 16:53:19 mds1 kernel: Mount-cache hash table entries: 256 Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys ns Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys cpuacct Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys memory Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys devices Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys freezer Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys net_cls Jan 4 16:53:19 mds1 kernel: Initializing cgroup subsys blkio Jan 4 16:53:19 mds1 kernel: mce: CPU supports 10 MCE banks Jan 4 16:53:19 mds1 kernel: alternatives: switching to unfair spinlock Jan 4 16:53:19 mds1 kernel: SMP alternatives: switching to UP code Jan 4 16:53:19 mds1 kernel: Freeing SMP alternatives: 33k freed Jan 4 16:53:19 mds1 kernel: ACPI: Core revision 20090903 Jan 4 16:53:19 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00 Jan 4 16:53:19 mds1 kernel: ftrace: allocating 20705 entries in 82 pages Jan 4 16:53:19 mds1 kernel: Setting APIC routing to flat Jan 4 16:53:19 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Jan 4 16:53:19 mds1 kernel: CPU0: Intel QEMU Virtual CPU version 0.14.0 stepping 03 Jan 4 16:53:19 mds1 kernel: Performance Events: unsupported p6 CPU model 2 no PMU driver, software events only. Jan 4 16:53:19 mds1 kernel: NMI watchdog disabled (cpu0): hardware events not enabled Jan 4 16:53:19 mds1 kernel: Brought up 1 CPUs Jan 4 16:53:19 mds1 kernel: Total of 1 processors activated (3191.76 BogoMIPS). Jan 4 16:53:19 mds1 kernel: devtmpfs: initialized Jan 4 16:53:19 mds1 kernel: regulator: core version 0.5 Jan 4 16:53:19 mds1 kernel: NET: Registered protocol family 16 Jan 4 16:53:19 mds1 kernel: ACPI: bus type pci registered Jan 4 16:53:19 mds1 kernel: PCI: Using configuration type 1 for base access Jan 4 16:53:19 mds1 kernel: bio: create slab at 0 Jan 4 16:53:19 mds1 kernel: ACPI: Interpreter enabled Jan 4 16:53:19 mds1 kernel: ACPI: (supports S0 S3 S4 S5) Jan 4 16:53:19 mds1 kernel: ACPI: Using IOAPIC for interrupt routing Jan 4 16:53:19 mds1 kernel: ACPI: No dock devices found. Jan 4 16:53:19 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) Jan 4 16:53:19 mds1 kernel: pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI Jan 4 16:53:19 mds1 kernel: pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0 Jan 4 16:53:19 mds1 kernel: HEST: Table is not found! Jan 4 16:53:19 mds1 kernel: vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none Jan 4 16:53:19 mds1 kernel: vgaarb: loaded Jan 4 16:53:19 mds1 kernel: SCSI subsystem initialized Jan 4 16:53:19 mds1 kernel: usbcore: registered new interface driver usbfs Jan 4 16:53:19 mds1 kernel: usbcore: registered new interface driver hub Jan 4 16:53:19 mds1 kernel: usbcore: registered new device driver usb Jan 4 16:53:19 mds1 kernel: PCI: Using ACPI for IRQ routing Jan 4 16:53:19 mds1 kernel: NetLabel: Initializing Jan 4 16:53:19 mds1 kernel: NetLabel: domain hash size = 128 Jan 4 16:53:19 mds1 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Jan 4 16:53:19 mds1 kernel: NetLabel: unlabeled traffic allowed by default Jan 4 16:53:19 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 Jan 4 16:53:19 mds1 kernel: hpet0: 3 comparators, 64-bit 100.000000 MHz counter Jan 4 16:53:19 mds1 kernel: Switching to clocksource kvm-clock Jan 4 16:53:19 mds1 kernel: pnp: PnP ACPI init Jan 4 16:53:19 mds1 kernel: ACPI: bus type pnp registered Jan 4 16:53:19 mds1 kernel: pnp: PnP ACPI: found 6 devices Jan 4 16:53:19 mds1 kernel: ACPI: ACPI bus type pnp unregistered Jan 4 16:53:19 mds1 kernel: NET: Registered protocol family 2 Jan 4 16:53:19 mds1 kernel: IP route cache hash table entries: 32768 (order: 6, 262144 bytes) Jan 4 16:53:19 mds1 kernel: TCP established hash table entries: 131072 (order: 9, 2097152 bytes) Jan 4 16:53:19 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Jan 4 16:53:19 mds1 kernel: TCP: Hash tables configured (established 131072 bind 65536) Jan 4 16:53:19 mds1 kernel: TCP reno registered Jan 4 16:53:19 mds1 kernel: NET: Registered protocol family 1 Jan 4 16:53:19 mds1 kernel: pci 0000:00:00.0: Limiting direct PCI/PCI transfers Jan 4 16:53:19 mds1 kernel: pci 0000:00:01.0: PIIX3: Enabling Passive Release Jan 4 16:53:19 mds1 kernel: pci 0000:00:01.0: Activating ISA DMA hang workarounds Jan 4 16:53:19 mds1 kernel: Trying to unpack rootfs image as initramfs... Jan 4 16:53:19 mds1 kernel: Freeing initrd memory: 54798k freed Jan 4 16:53:19 mds1 kernel: audit: initializing netlink socket (disabled) Jan 4 16:53:19 mds1 kernel: type=2000 audit(1325713990.951:1): initialized Jan 4 16:53:19 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages Jan 4 16:53:19 mds1 kernel: VFS: Disk quotas dquot_6.5.2 Jan 4 16:53:19 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Jan 4 16:53:19 mds1 kernel: msgmni has been set to 1987 Jan 4 16:53:19 mds1 kernel: alg: No test for stdrng (krng) Jan 4 16:53:19 mds1 kernel: ksign: Installing public key data Jan 4 16:53:19 mds1 kernel: Loading keyring Jan 4 16:53:19 mds1 kernel: - Added public key CBA5A3F3E494381 Jan 4 16:53:19 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key) Jan 4 16:53:19 mds1 kernel: - Added public key D4A26C9CCD09BEDA Jan 4 16:53:19 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program Jan 4 16:53:19 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) Jan 4 16:53:19 mds1 kernel: io scheduler noop registered Jan 4 16:53:19 mds1 kernel: io scheduler anticipatory registered Jan 4 16:53:19 mds1 kernel: io scheduler deadline registered (default) Jan 4 16:53:19 mds1 kernel: io scheduler cfq registered Jan 4 16:53:19 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Jan 4 16:53:19 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Jan 4 16:53:19 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [1] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [2] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [3] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [4] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [5] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [6] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [7] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [8] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [9] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [10] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [11] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [12] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [13] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [14] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [15] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [16] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [17] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [18] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [19] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [20] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [21] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [22] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [23] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [24] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [25] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [26] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [27] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [28] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [29] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [30] registered Jan 4 16:53:19 mds1 kernel: acpiphp: Slot [31] registered Jan 4 16:53:19 mds1 kernel: pci-stub: invalid id string "" Jan 4 16:53:19 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 Jan 4 16:53:19 mds1 kernel: ACPI: Power Button [PWRF] Jan 4 16:53:19 mds1 kernel: processor LNXCPU:00: registered as cooling_device0 Jan 4 16:53:19 mds1 kernel: ERST: Table is not found! Jan 4 16:53:19 mds1 kernel: Non-volatile memory driver v1.3 Jan 4 16:53:19 mds1 kernel: Linux agpgart interface v0.103 Jan 4 16:53:19 mds1 kernel: crash memory driver: version 1.1 Jan 4 16:53:19 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Jan 4 16:53:19 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jan 4 16:53:19 mds1 kernel: brd: module loaded Jan 4 16:53:19 mds1 kernel: loop: module loaded Jan 4 16:53:19 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1 Jan 4 16:53:19 mds1 kernel: Fixed MDIO Bus: probed Jan 4 16:53:19 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Jan 4 16:53:19 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Jan 4 16:53:19 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 Jan 4 16:53:19 mds1 kernel: uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 Jan 4 16:53:19 mds1 kernel: uhci_hcd 0000:00:01.2: UHCI Host Controller Jan 4 16:53:19 mds1 kernel: uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 Jan 4 16:53:19 mds1 kernel: uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 Jan 4 16:53:19 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 Jan 4 16:53:19 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jan 4 16:53:19 mds1 kernel: usb usb1: Product: UHCI Host Controller Jan 4 16:53:19 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 uhci_hcd Jan 4 16:53:19 mds1 kernel: usb usb1: SerialNumber: 0000:00:01.2 Jan 4 16:53:19 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice Jan 4 16:53:19 mds1 kernel: hub 1-0:1.0: USB hub found Jan 4 16:53:19 mds1 kernel: hub 1-0:1.0: 2 ports detected Jan 4 16:53:19 mds1 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 Jan 4 16:53:19 mds1 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Jan 4 16:53:19 mds1 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Jan 4 16:53:19 mds1 kernel: mice: PS/2 mouse device common for all mice Jan 4 16:53:19 mds1 kernel: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 Jan 4 16:53:19 mds1 kernel: rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 Jan 4 16:53:19 mds1 kernel: rtc0: alarms up to one day, 114 bytes nvram, hpet irqs Jan 4 16:53:19 mds1 kernel: cpuidle: using governor ladder Jan 4 16:53:19 mds1 kernel: cpuidle: using governor menu Jan 4 16:53:19 mds1 kernel: usbcore: registered new interface driver hiddev Jan 4 16:53:19 mds1 kernel: usbcore: registered new interface driver usbhid Jan 4 16:53:19 mds1 kernel: usbhid: v2.6:USB HID core driver Jan 4 16:53:19 mds1 kernel: TCP cubic registered Jan 4 16:53:19 mds1 kernel: Initializing XFRM netlink socket Jan 4 16:53:19 mds1 kernel: NET: Registered protocol family 17 Jan 4 16:53:19 mds1 kernel: registered taskstats version 1 Jan 4 16:53:19 mds1 kernel: rtc_cmos 00:01: setting system clock to 2012-01-04 21:53:09 UTC (1325713989) Jan 4 16:53:19 mds1 kernel: Initalizing network drop monitor service Jan 4 16:53:19 mds1 kernel: Freeing unused kernel memory: 1232k freed Jan 4 16:53:19 mds1 kernel: Write protecting the kernel read-only data: 10240k Jan 4 16:53:19 mds1 kernel: Freeing unused kernel memory: 1108k freed Jan 4 16:53:19 mds1 kernel: Freeing unused kernel memory: 1796k freed Jan 4 16:53:19 mds1 kernel: dracut: dracut-004-32.el6 Jan 4 16:53:19 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation Jan 4 16:53:19 mds1 kernel: device-mapper: uevent: version 1.0.3 Jan 4 16:53:19 mds1 kernel: device-mapper: ioctl: 4.20.6-ioctl (2011-02-02) initialised: dm-devel@redhat.com Jan 4 16:53:19 mds1 kernel: udev: starting version 147 Jan 4 16:53:19 mds1 kernel: dracut: Starting plymouth daemon Jan 4 16:53:19 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation Jan 4 16:53:19 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation Jan 4 16:53:19 mds1 kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 Jan 4 16:53:19 mds1 kernel: virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10 Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 Jan 4 16:53:19 mds1 kernel: virtio-pci 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 10 (level, high) -> IRQ 10 Jan 4 16:53:19 mds1 kernel: virtio-pci 0000:00:07.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10 Jan 4 16:53:19 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 Jan 4 16:53:19 mds1 kernel: sym53c8xx 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, high) -> IRQ 11 Jan 4 16:53:19 mds1 kernel: sym0: <895a> rev 0x0 at pci 0000:00:05.0 irq 11 Jan 4 16:53:19 mds1 kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking Jan 4 16:53:19 mds1 kernel: sym0: SCSI BUS has been reset. Jan 4 16:53:19 mds1 kernel: scsi0 : sym-2.2.3 Jan 4 16:53:19 mds1 kernel: scsi1 : ata_piix Jan 4 16:53:19 mds1 kernel: scsi2 : ata_piix Jan 4 16:53:19 mds1 kernel: ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 Jan 4 16:53:19 mds1 kernel: ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 Jan 4 16:53:19 mds1 kernel: vda: vda1 vda2 Jan 4 16:53:19 mds1 kernel: scsi: waiting for bus probes to complete ... Jan 4 16:53:19 mds1 kernel: sym0: unknown interrupt(s) ignored, ISTAT=0x5 DSTAT=0x80 SIST=0x0 Jan 4 16:53:19 mds1 kernel: scsi 0:0:0:0: Direct-Access QEMU QEMU HARDDISK 0.14 PQ: 0 ANSI: 5 Jan 4 16:53:19 mds1 kernel: scsi target0:0:0: tagged command queuing enabled, command queue depth 16. Jan 4 16:53:19 mds1 kernel: scsi target0:0:0: Beginning Domain Validation Jan 4 16:53:19 mds1 kernel: scsi target0:0:0: Domain Validation skipping write tests Jan 4 16:53:19 mds1 kernel: scsi target0:0:0: Ending Domain Validation Jan 4 16:53:19 mds1 kernel: scsi 0:0:1:0: Direct-Access QEMU QEMU HARDDISK 0.14 PQ: 0 ANSI: 5 Jan 4 16:53:19 mds1 kernel: scsi target0:0:1: tagged command queuing enabled, command queue depth 16. Jan 4 16:53:19 mds1 kernel: scsi target0:0:1: Beginning Domain Validation Jan 4 16:53:19 mds1 kernel: scsi target0:0:1: Domain Validation skipping write tests Jan 4 16:53:19 mds1 kernel: scsi target0:0:1: Ending Domain Validation Jan 4 16:53:19 mds1 kernel: sd 0:0:0:0: [sda] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB) Jan 4 16:53:19 mds1 kernel: sd 0:0:0:0: [sda] Write Protect is off Jan 4 16:53:19 mds1 kernel: sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 4 16:53:19 mds1 kernel: sd 0:0:1:0: [sdb] 204800 512-byte logical blocks: (104 MB/100 MiB) Jan 4 16:53:19 mds1 kernel: sd 0:0:1:0: [sdb] Write Protect is off Jan 4 16:53:19 mds1 kernel: sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 4 16:53:19 mds1 kernel: sda: Jan 4 16:53:19 mds1 kernel: sdb: unknown partition table Jan 4 16:53:19 mds1 kernel: sd 0:0:0:0: [sda] Attached SCSI disk Jan 4 16:53:19 mds1 kernel: unknown partition table Jan 4 16:53:19 mds1 kernel: sd 0:0:1:0: [sdb] Attached SCSI disk Jan 4 16:53:19 mds1 kernel: dracut: Scanning devices vda2 for LVM logical volumes LustreVG/root LustreVG/swap Jan 4 16:53:19 mds1 kernel: dracut: inactive '/dev/LustreVG/root' [1.37 GiB] inherit Jan 4 16:53:19 mds1 kernel: dracut: inactive '/dev/LustreVG/swap' [544.00 MiB] inherit Jan 4 16:53:19 mds1 kernel: EXT3-fs (dm-0): recovery required on readonly filesystem Jan 4 16:53:19 mds1 kernel: EXT3-fs (dm-0): write access will be enabled during recovery Jan 4 16:53:19 mds1 kernel: kjournald starting. Commit interval 5 seconds Jan 4 16:53:19 mds1 kernel: EXT3-fs (dm-0): recovery complete Jan 4 16:53:19 mds1 kernel: EXT3-fs (dm-0): mounted filesystem with ordered data mode Jan 4 16:53:19 mds1 kernel: dracut: Mounted root filesystem /dev/mapper/LustreVG-root Jan 4 16:53:19 mds1 kernel: dracut: Loading SELinux policy Jan 4 16:53:19 mds1 kernel: SELinux: Disabled at runtime. Jan 4 16:53:19 mds1 kernel: type=1404 audit(1325713994.402:2): selinux=0 auid=4294967295 ses=4294967295 Jan 4 16:53:19 mds1 kernel: dracut: /sbin/load_policy: Can't load policy: No such file or directory Jan 4 16:53:19 mds1 kernel: dracut: Switching root Jan 4 16:53:19 mds1 kernel: udev: starting version 147 Jan 4 16:53:19 mds1 kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 Jan 4 16:53:19 mds1 kernel: Intel ICH 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 Jan 4 16:53:19 mds1 kernel: intel8x0_measure_ac97_clock: measured 50925 usecs (4467 samples) Jan 4 16:53:19 mds1 kernel: intel8x0: measured clock 87717 rejected Jan 4 16:53:19 mds1 kernel: intel8x0_measure_ac97_clock: measured 50927 usecs (4467 samples) Jan 4 16:53:19 mds1 kernel: intel8x0: measured clock 87713 rejected Jan 4 16:53:19 mds1 kernel: intel8x0_measure_ac97_clock: measured 51042 usecs (4504 samples) Jan 4 16:53:19 mds1 kernel: intel8x0: measured clock 88241 rejected Jan 4 16:53:19 mds1 kernel: intel8x0: clocking to 48000 Jan 4 16:53:19 mds1 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0 Jan 4 16:53:19 mds1 kernel: sd 0:0:1:0: Attached scsi generic sg1 type 0 Jan 4 16:53:19 mds1 kernel: EXT3-fs (dm-0): using internal journal Jan 4 16:53:19 mds1 kernel: kjournald starting. Commit interval 5 seconds Jan 4 16:53:19 mds1 kernel: EXT3-fs (vda1): using internal journal Jan 4 16:53:19 mds1 kernel: EXT3-fs (vda1): mounted filesystem with ordered data mode Jan 4 16:53:19 mds1 kernel: Adding 557048k swap on /dev/mapper/LustreVG-swap. Priority:-1 extents:1 across:557048k Jan 4 16:53:19 mds1 kernel: microcode: CPU0 sig=0x623, pf=0x0, revision=0x0 Jan 4 16:53:19 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-02-03 Jan 4 16:53:19 mds1 kernel: Microcode Update Driver: v2.00 , Peter Oruba Jan 4 16:53:19 mds1 kernel: Microcode Update Driver: v2.00 removed. Jan 4 16:53:19 mds1 kernel: NET: Registered protocol family 10 Jan 4 16:53:19 mds1 kernel: lo: Disabled Privacy Extensions Jan 4 16:53:19 mds1 rpc.statd[1251]: Version 1.2.2 starting Jan 4 16:53:19 mds1 sm-notify[1252]: Version 1.2.2 starting Jan 4 16:53:20 mds1 kernel: RPC: Registered udp transport module. Jan 4 16:53:20 mds1 kernel: RPC: Registered tcp transport module. Jan 4 16:53:20 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Jan 4 16:53:20 mds1 corosync[1317]: [MAIN ] Corosync Cluster Engine ('1.2.3'): started and ready to provide service. Jan 4 16:53:20 mds1 corosync[1317]: [MAIN ] Corosync built-in features: nss rdma Jan 4 16:53:20 mds1 corosync[1317]: [MAIN ] Successfully read main configuration file '/etc/corosync/corosync.conf'. Jan 4 16:53:20 mds1 corosync[1317]: [TOTEM ] Initializing transport (UDP/IP). Jan 4 16:53:20 mds1 corosync[1317]: [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0). Jan 4 16:53:20 mds1 corosync[1317]: [TOTEM ] The network interface [192.168.122.89] is now up. Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: process_ais_conf: Reading configure Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: config_find_init: Local handle: 2013064636357672962 for logging Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: config_find_next: Processing additional logging options... Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: Found 'off' for option: debug Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: Found 'no' for option: to_logfile Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: Found 'yes' for option: to_syslog Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: config_find_init: Local handle: 4730966301143465987 for quorum Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: config_find_next: No additional configuration supplied for: quorum Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: No default for option: provider Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: config_find_init: Local handle: 7739444317642555396 for service Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: config_find_next: Processing additional service options... Jan 4 16:53:20 mds1 lrmd: [1325]: info: G_main_add_SignalHandler: Added signal handler for signal 15 Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: Invoked: /usr/lib64/heartbeat/stonithd Jan 4 16:53:20 mds1 cib: [1324]: info: Invoked: /usr/lib64/heartbeat/cib Jan 4 16:53:20 mds1 attrd: [1326]: info: Invoked: /usr/lib64/heartbeat/attrd Jan 4 16:53:20 mds1 pengine: [1327]: info: Invoked: /usr/lib64/heartbeat/pengine Jan 4 16:53:20 mds1 crmd: [1328]: info: Invoked: /usr/lib64/heartbeat/crmd Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: Defaulting to '0' for option: ver Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/root Jan 4 16:53:20 mds1 cib: [1324]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Jan 4 16:53:20 mds1 attrd: [1326]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Jan 4 16:53:20 mds1 pengine: [1327]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Jan 4 16:53:20 mds1 lrmd: [1325]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Jan 4 16:53:20 mds1 crmd: [1328]: info: crm_log_init_worker: Changed active directory to /var/lib/heartbeat/cores/hacluster Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: Defaulting to 'pcmk' for option: clustername Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Jan 4 16:53:20 mds1 cib: [1324]: info: G_main_add_TriggerHandler: Added signal manual handler Jan 4 16:53:20 mds1 attrd: [1326]: info: main: Starting up Jan 4 16:53:20 mds1 pengine: [1327]: info: main: Starting pengine Jan 4 16:53:20 mds1 lrmd: [1325]: info: enabling coredumps Jan 4 16:53:20 mds1 crmd: [1328]: info: main: CRM Hg Version: f059ec7ced7a86f18e5490b67ebf4a0b963bccfe Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_logd Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: get_cluster_type: Cluster type is: 'openais'. Jan 4 16:53:20 mds1 cib: [1324]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Jan 4 16:53:20 mds1 attrd: [1326]: info: get_cluster_type: Cluster type is: 'openais'. Jan 4 16:53:20 mds1 lrmd: [1325]: info: G_main_add_SignalHandler: Added signal handler for signal 10 Jan 4 16:53:20 mds1 crmd: [1328]: info: crmd_init: Starting crmd Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: get_config_opt: Defaulting to 'no' for option: use_mgmtd Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Jan 4 16:53:20 mds1 cib: [1324]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest: /var/lib/heartbeat/crm/cib.xml.sig) Jan 4 16:53:20 mds1 attrd: [1326]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Jan 4 16:53:20 mds1 lrmd: [1325]: info: G_main_add_SignalHandler: Added signal handler for signal 12 Jan 4 16:53:20 mds1 crmd: [1328]: info: G_main_add_SignalHandler: Added signal handler for signal 17 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_startup: CRM: Initialized Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Jan 4 16:53:20 mds1 attrd: [1326]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Jan 4 16:53:20 mds1 lrmd: [1325]: info: Started. Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] Logging: Initialized pcmk_startup Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: init_ais_connection_classic: AIS connection established Jan 4 16:53:20 mds1 attrd: [1326]: info: init_ais_connection_classic: AIS connection established Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_startup: Maximum core file size is: 18446744073709551615 Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: get_ais_nodeid: Server details: id=1501210816 uname=mds1 cname=pcmk Jan 4 16:53:20 mds1 attrd: [1326]: info: get_ais_nodeid: Server details: id=1501210816 uname=mds1 cname=pcmk Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_startup: Service: 10 Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Jan 4 16:53:20 mds1 attrd: [1326]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_startup: Local hostname: mds1 Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: crm_new_peer: Node mds1 now has id: 1501210816 Jan 4 16:53:20 mds1 attrd: [1326]: info: crm_new_peer: Node mds1 now has id: 1501210816 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_update_nodeid: Local node id: 1501210816 Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: crm_new_peer: Node 1501210816 is now known as mds1 Jan 4 16:53:20 mds1 attrd: [1326]: info: crm_new_peer: Node 1501210816 is now known as mds1 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: update_member: Creating entry for node 1501210816 born on 0 Jan 4 16:53:20 mds1 stonith-ng: [1323]: info: main: Starting stonith-ng mainloop Jan 4 16:53:20 mds1 attrd: [1326]: info: main: Cluster connection active Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: update_member: 0x14091c0 Node 1501210816 now known as mds1 (was: (null)) Jan 4 16:53:20 mds1 attrd: [1326]: info: main: Accepting attribute updates Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: update_member: Node mds1 now has 1 quorum votes (was 0) Jan 4 16:53:20 mds1 attrd: [1326]: info: main: Starting mainloop... Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: update_member: Node 1501210816/mds1 is now: member Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: spawn_child: Forked child 1323 for process stonith-ng Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: spawn_child: Forked child 1324 for process cib Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: spawn_child: Forked child 1325 for process lrmd Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: spawn_child: Forked child 1326 for process attrd Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: spawn_child: Forked child 1327 for process pengine Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: spawn_child: Forked child 1328 for process crmd Jan 4 16:53:20 mds1 corosync[1317]: [SERV ] Service engine loaded: Pacemaker Cluster Manager 1.1.2 Jan 4 16:53:20 mds1 corosync[1317]: [SERV ] Service engine loaded: corosync extended virtual synchrony service Jan 4 16:53:20 mds1 corosync[1317]: [SERV ] Service engine loaded: corosync configuration service Jan 4 16:53:20 mds1 corosync[1317]: [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 Jan 4 16:53:20 mds1 corosync[1317]: [SERV ] Service engine loaded: corosync cluster config database access v1.01 Jan 4 16:53:20 mds1 corosync[1317]: [SERV ] Service engine loaded: corosync profile loading service Jan 4 16:53:20 mds1 corosync[1317]: [SERV ] Service engine loaded: corosync cluster quorum service v0.1 Jan 4 16:53:20 mds1 corosync[1317]: [MAIN ] Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine. Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5056: memb=0, new=0, lost=0 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5056: memb=1, new=1, lost=0 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: NEW: mds1 1501210816 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: update_member: Node mds1 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 16:53:20 mds1 corosync[1317]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 16:53:20 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:20 mds1 corosync[1317]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 16:53:20 mds1 corosync[1317]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_ipc: Recorded connection 0x14124c0 for stonith-ng/1323 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_ipc: Recorded connection 0x1416820 for attrd/1326 Jan 4 16:53:20 mds1 cib: [1324]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-82.raw Jan 4 16:53:20 mds1 cib: [1324]: info: write_cib_contents: Wrote version 0.248.0 of the CIB to disk (digest: d31a68f2ed6cfeb15baa67fbce55e235) Jan 4 16:53:20 mds1 cib: [1324]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.noQl3t (digest: /var/lib/heartbeat/crm/cib.rxcv92) Jan 4 16:53:20 mds1 cib: [1324]: info: startCib: CIB Initialization completed successfully Jan 4 16:53:20 mds1 cib: [1324]: info: get_cluster_type: Cluster type is: 'openais'. Jan 4 16:53:20 mds1 cib: [1324]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Jan 4 16:53:20 mds1 cib: [1324]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Jan 4 16:53:20 mds1 cib: [1324]: info: init_ais_connection_classic: AIS connection established Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_ipc: Recorded connection 0x141b140 for cib/1324 Jan 4 16:53:20 mds1 corosync[1317]: [pcmk ] info: pcmk_ipc: Sending membership update 5056 to cib Jan 4 16:53:20 mds1 cib: [1324]: info: get_ais_nodeid: Server details: id=1501210816 uname=mds1 cname=pcmk Jan 4 16:53:20 mds1 cib: [1324]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Jan 4 16:53:20 mds1 cib: [1324]: info: crm_new_peer: Node mds1 now has id: 1501210816 Jan 4 16:53:20 mds1 cib: [1324]: info: crm_new_peer: Node 1501210816 is now known as mds1 Jan 4 16:53:20 mds1 cib: [1324]: info: cib_init: Starting cib mainloop Jan 4 16:53:20 mds1 cib: [1324]: info: ais_dispatch_message: Membership 5056: quorum still lost Jan 4 16:53:20 mds1 cib: [1324]: info: crm_update_peer: Node mds1: id=1501210816 state=member (new) addr=r(0) ip(192.168.122.89) (new) votes=1 (new) born=0 seen=5056 proc=00000000000000000000000000111312 (new) Jan 4 16:53:21 mds1 crmd: [1328]: info: do_cib_control: CIB connection established Jan 4 16:53:21 mds1 crmd: [1328]: info: get_cluster_type: Cluster type is: 'openais'. Jan 4 16:53:21 mds1 crmd: [1328]: info: crm_cluster_connect: Connecting to cluster infrastructure: classic openais (with plugin) Jan 4 16:53:21 mds1 crmd: [1328]: info: init_ais_connection_classic: Creating connection to our Corosync plugin Jan 4 16:53:21 mds1 crmd: [1328]: info: init_ais_connection_classic: AIS connection established Jan 4 16:53:21 mds1 corosync[1317]: [pcmk ] info: pcmk_ipc: Recorded connection 0x141f940 for crmd/1328 Jan 4 16:53:21 mds1 corosync[1317]: [pcmk ] info: pcmk_ipc: Sending membership update 5056 to crmd Jan 4 16:53:21 mds1 crmd: [1328]: info: get_ais_nodeid: Server details: id=1501210816 uname=mds1 cname=pcmk Jan 4 16:53:21 mds1 crmd: [1328]: info: init_ais_connection_once: Connection to 'classic openais (with plugin)': established Jan 4 16:53:21 mds1 crmd: [1328]: info: crm_new_peer: Node mds1 now has id: 1501210816 Jan 4 16:53:21 mds1 crmd: [1328]: info: crm_new_peer: Node 1501210816 is now known as mds1 Jan 4 16:53:21 mds1 crmd: [1328]: info: ais_status_callback: status: mds1 is now unknown Jan 4 16:53:21 mds1 crmd: [1328]: info: do_ha_control: Connected to the cluster Jan 4 16:53:21 mds1 crmd: [1328]: info: do_started: Delaying start, no membership data (0000000000100000) Jan 4 16:53:21 mds1 crmd: [1328]: info: crmd_init: Starting crmd's mainloop Jan 4 16:53:21 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5056: quorum still lost Jan 4 16:53:21 mds1 crmd: [1328]: notice: crmd_peer_update: Status update: Client mds1/crmd now has status [online] (DC=) Jan 4 16:53:21 mds1 crmd: [1328]: info: ais_status_callback: status: mds1 is now member (was unknown) Jan 4 16:53:21 mds1 crmd: [1328]: info: crm_update_peer: Node mds1: id=1501210816 state=member (new) addr=r(0) ip(192.168.122.89) (new) votes=1 (new) born=0 seen=5056 proc=00000000000000000000000000111312 (new) Jan 4 16:53:21 mds1 crmd: [1328]: info: do_started: Delaying start, Config not read (0000000000000040) Jan 4 16:53:21 mds1 crmd: [1328]: info: config_query_callback: Checking for expired actions every 900000ms Jan 4 16:53:21 mds1 crmd: [1328]: info: config_query_callback: Sending expected-votes=4 to corosync Jan 4 16:53:21 mds1 crmd: [1328]: info: do_started: The local CRM is operational Jan 4 16:53:21 mds1 corosync[1317]: [pcmk ] info: update_expected_votes: Expected quorum votes 2 -> 4 Jan 4 16:53:21 mds1 crmd: [1328]: info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ] Jan 4 16:53:22 mds1 avahi-daemon[1355]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70). Jan 4 16:53:22 mds1 avahi-daemon[1355]: Successfully dropped root privileges. Jan 4 16:53:22 mds1 avahi-daemon[1355]: avahi-daemon 0.6.25 starting up. Jan 4 16:53:22 mds1 avahi-daemon[1355]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Jan 4 16:53:22 mds1 avahi-daemon[1355]: Successfully called chroot(). Jan 4 16:53:22 mds1 avahi-daemon[1355]: Successfully dropped remaining capabilities. Jan 4 16:53:22 mds1 avahi-daemon[1355]: Loading service file /services/ssh.service. Jan 4 16:53:22 mds1 avahi-daemon[1355]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.122.89. Jan 4 16:53:22 mds1 avahi-daemon[1355]: New relevant interface eth0.IPv4 for mDNS. Jan 4 16:53:22 mds1 avahi-daemon[1355]: Network interface enumeration completed. Jan 4 16:53:22 mds1 avahi-daemon[1355]: Registering new address record for fe80::5054:ff:fed9:98ec on eth0.*. Jan 4 16:53:22 mds1 avahi-daemon[1355]: Registering new address record for 192.168.122.89 on eth0.IPv4. Jan 4 16:53:22 mds1 avahi-daemon[1355]: Registering HINFO record with values 'X86_64'/'LINUX'. Jan 4 16:53:22 mds1 kernel: TECH PREVIEW: fscache may not be fully supported. Jan 4 16:53:22 mds1 kernel: Please review provided documentation for limitations. Jan 4 16:53:22 mds1 kernel: Disabling lock debugging due to kernel taint Jan 4 16:53:22 mds1 kernel: Slow work thread pool: Starting up Jan 4 16:53:22 mds1 kernel: Slow work thread pool: Ready Jan 4 16:53:22 mds1 kernel: FS-Cache: Loaded Jan 4 16:53:22 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5056: quorum still lost Jan 4 16:53:22 mds1 crmd: [1328]: info: te_connect_stonith: Attempting connection to fencing daemon... Jan 4 16:53:22 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching Jan 4 16:53:22 mds1 acpid: starting up Jan 4 16:53:22 mds1 acpid: 1 rule loaded Jan 4 16:53:22 mds1 acpid: waiting for events: event logging is off Jan 4 16:53:22 mds1 acpid: client connected from 1438[68:68] Jan 4 16:53:22 mds1 acpid: 1 client rule loaded Jan 4 16:53:23 mds1 avahi-daemon[1355]: Server startup complete. Host name is mds1.local. Local service cookie is 2864098762. Jan 4 16:53:23 mds1 crmd: [1328]: info: te_connect_stonith: Connected Jan 4 16:53:23 mds1 avahi-dnsconfd[1580]: Successfully connected to Avahi daemon. Jan 4 16:53:24 mds1 avahi-daemon[1355]: Service "mds1" (/services/ssh.service) successfully established. Jan 4 16:53:25 mds1 attrd: [1326]: info: cib_connect: Connected to the CIB after 1 signon attempts Jan 4 16:53:25 mds1 attrd: [1326]: info: cib_connect: Sending full refresh Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5072: memb=1, new=0, lost=0 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5072: memb=2, new=1, lost=0 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: update_member: Creating entry for node 2608507072 born on 5072 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: update_member: Node 2608507072/unknown is now: member Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2608507072 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2608507072 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: send_member_notification: Sending membership update 5072 to 2 children Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: update_member: 0x14091c0 Node 1501210816 ((null)) born on: 5072 Jan 4 16:53:31 mds1 corosync[1317]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 16:53:31 mds1 cib: [1324]: info: ais_dispatch_message: Membership 5072: quorum still lost Jan 4 16:53:31 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5072: quorum still lost Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: update_member: 0x14097b0 Node 2608507072 (mds2) born on: 5072 Jan 4 16:53:31 mds1 cib: [1324]: info: crm_new_peer: Node now has id: 2608507072 Jan 4 16:53:31 mds1 crmd: [1328]: info: crm_new_peer: Node now has id: 2608507072 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: update_member: 0x14097b0 Node 2608507072 now known as mds2 (was: (null)) Jan 4 16:53:31 mds1 cib: [1324]: info: crm_update_peer: Node (null): id=2608507072 state=member (new) addr=r(0) ip(192.168.122.155) votes=0 born=0 seen=5072 proc=00000000000000000000000000000000 Jan 4 16:53:31 mds1 crmd: [1328]: info: crm_update_peer: Node (null): id=2608507072 state=member (new) addr=r(0) ip(192.168.122.155) votes=0 born=0 seen=5072 proc=00000000000000000000000000000000 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: update_member: Node mds2 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 16:53:31 mds1 cib: [1324]: info: ais_dispatch_message: Membership 5072: quorum still lost Jan 4 16:53:31 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5072: quorum still lost Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: update_member: Node mds2 now has 1 quorum votes (was 0) Jan 4 16:53:31 mds1 cib: [1324]: info: crm_get_peer: Node 2608507072 is now known as mds2 Jan 4 16:53:31 mds1 crmd: [1328]: info: crm_get_peer: Node 2608507072 is now known as mds2 Jan 4 16:53:31 mds1 corosync[1317]: [pcmk ] info: send_member_notification: Sending membership update 5072 to 2 children Jan 4 16:53:31 mds1 cib: [1324]: info: crm_update_peer: Node mds2: id=2608507072 state=member addr=r(0) ip(192.168.122.155) votes=1 (new) born=5072 seen=5072 proc=00000000000000000000000000111312 (new) Jan 4 16:53:31 mds1 crmd: [1328]: info: ais_status_callback: status: mds2 is now member Jan 4 16:53:31 mds1 crmd: [1328]: notice: crmd_peer_update: Status update: Client mds2/crmd now has status [online] (DC=) Jan 4 16:53:31 mds1 crmd: [1328]: info: crm_update_peer: Node mds2: id=2608507072 state=member addr=r(0) ip(192.168.122.155) votes=1 (new) born=5072 seen=5072 proc=00000000000000000000000000111312 (new) Jan 4 16:53:31 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:31 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:31 mds1 corosync[1317]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 16:53:31 mds1 corosync[1317]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5076: memb=2, new=0, lost=0 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: memb: mds2 2608507072 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5076: memb=3, new=1, lost=0 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: update_member: Creating entry for node 2692393152 born on 5076 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: update_member: Node 2692393152/unknown is now: member Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2692393152 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: mds2 2608507072 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2692393152 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: send_member_notification: Sending membership update 5076 to 2 children Jan 4 16:53:32 mds1 corosync[1317]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 16:53:32 mds1 cib: [1324]: info: ais_dispatch_message: Membership 5076: quorum still lost Jan 4 16:53:32 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5076: quorum still lost Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: update_member: 0x141ab80 Node 2692393152 (oss2) born on: 5076 Jan 4 16:53:32 mds1 cib: [1324]: info: crm_new_peer: Node now has id: 2692393152 Jan 4 16:53:32 mds1 crmd: [1328]: info: crm_new_peer: Node now has id: 2692393152 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: update_member: 0x141ab80 Node 2692393152 now known as oss2 (was: (null)) Jan 4 16:53:32 mds1 cib: [1324]: info: crm_update_peer: Node (null): id=2692393152 state=member (new) addr=r(0) ip(192.168.122.160) votes=0 born=0 seen=5076 proc=00000000000000000000000000000000 Jan 4 16:53:32 mds1 crmd: [1328]: info: crm_update_peer: Node (null): id=2692393152 state=member (new) addr=r(0) ip(192.168.122.160) votes=0 born=0 seen=5076 proc=00000000000000000000000000000000 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: update_member: Node oss2 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 16:53:32 mds1 cib: [1324]: notice: ais_dispatch_message: Membership 5076: quorum acquired Jan 4 16:53:32 mds1 crmd: [1328]: notice: ais_dispatch_message: Membership 5076: quorum acquired Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: update_member: Node oss2 now has 1 quorum votes (was 0) Jan 4 16:53:32 mds1 cib: [1324]: info: crm_get_peer: Node 2692393152 is now known as oss2 Jan 4 16:53:32 mds1 crmd: [1328]: info: crm_get_peer: Node 2692393152 is now known as oss2 Jan 4 16:53:32 mds1 corosync[1317]: [pcmk ] info: send_member_notification: Sending membership update 5076 to 2 children Jan 4 16:53:32 mds1 cib: [1324]: info: crm_update_peer: Node oss2: id=2692393152 state=member addr=r(0) ip(192.168.122.160) votes=1 (new) born=5076 seen=5076 proc=00000000000000000000000000111312 (new) Jan 4 16:53:32 mds1 crmd: [1328]: info: ais_status_callback: status: oss2 is now member Jan 4 16:53:32 mds1 crmd: [1328]: notice: crmd_peer_update: Status update: Client oss2/crmd now has status [online] (DC=) Jan 4 16:53:32 mds1 crmd: [1328]: info: crm_update_peer: Node oss2: id=2692393152 state=member addr=r(0) ip(192.168.122.160) votes=1 (new) born=5076 seen=5076 proc=00000000000000000000000000111312 (new) Jan 4 16:53:32 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:32 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:32 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:32 mds1 corosync[1317]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 16:53:32 mds1 corosync[1317]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 5080: memb=3, new=0, lost=0 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: memb: mds1 1501210816 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: memb: mds2 2608507072 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: memb: oss2 2692393152 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 5080: memb=4, new=1, lost=0 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: update_member: Creating entry for node 2474289344 born on 5080 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: update_member: Node 2474289344/unknown is now: member Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: NEW: .pending. 2474289344 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: mds1 1501210816 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: .pending. 2474289344 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: mds2 2608507072 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: pcmk_peer_update: MEMB: oss2 2692393152 Jan 4 16:53:33 mds1 cib: [1324]: info: ais_dispatch_message: Membership 5080: quorum retained Jan 4 16:53:33 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5080: quorum retained Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: send_member_notification: Sending membership update 5080 to 2 children Jan 4 16:53:33 mds1 cib: [1324]: info: crm_new_peer: Node now has id: 2474289344 Jan 4 16:53:33 mds1 crmd: [1328]: info: crm_new_peer: Node now has id: 2474289344 Jan 4 16:53:33 mds1 corosync[1317]: [TOTEM ] A processor joined or left the membership and a new membership was formed. Jan 4 16:53:33 mds1 cib: [1324]: info: crm_update_peer: Node (null): id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=0 born=0 seen=5080 proc=00000000000000000000000000000000 Jan 4 16:53:33 mds1 crmd: [1328]: info: crm_update_peer: Node (null): id=2474289344 state=member (new) addr=r(0) ip(192.168.122.147) votes=0 born=0 seen=5080 proc=00000000000000000000000000000000 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: update_member: 0x1426b40 Node 2474289344 (oss1) born on: 5080 Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: update_member: 0x1426b40 Node 2474289344 now known as oss1 (was: (null)) Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: update_member: Node oss1 now has process list: 00000000000000000000000000111312 (1118994) Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: update_member: Node oss1 now has 1 quorum votes (was 0) Jan 4 16:53:33 mds1 corosync[1317]: [pcmk ] info: send_member_notification: Sending membership update 5080 to 2 children Jan 4 16:53:33 mds1 cib: [1324]: info: ais_dispatch_message: Membership 5080: quorum retained Jan 4 16:53:33 mds1 cib: [1324]: info: crm_get_peer: Node 2474289344 is now known as oss1 Jan 4 16:53:33 mds1 cib: [1324]: info: crm_update_peer: Node oss1: id=2474289344 state=member addr=r(0) ip(192.168.122.147) votes=1 (new) born=5080 seen=5080 proc=00000000000000000000000000111312 (new) Jan 4 16:53:33 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5080: quorum retained Jan 4 16:53:33 mds1 crmd: [1328]: info: crm_get_peer: Node 2474289344 is now known as oss1 Jan 4 16:53:33 mds1 crmd: [1328]: info: ais_status_callback: status: oss1 is now member Jan 4 16:53:33 mds1 crmd: [1328]: notice: crmd_peer_update: Status update: Client oss1/crmd now has status [online] (DC=) Jan 4 16:53:33 mds1 crmd: [1328]: info: crm_update_peer: Node oss1: id=2474289344 state=member addr=r(0) ip(192.168.122.147) votes=1 (new) born=5080 seen=5080 proc=00000000000000000000000000111312 (new) Jan 4 16:53:33 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:33 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:33 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:33 mds1 corosync[1317]: [CPG ] downlist received left_list: 0 Jan 4 16:53:33 mds1 corosync[1317]: [CPG ] chosen downlist from node r(0) ip(192.168.122.89) Jan 4 16:53:33 mds1 corosync[1317]: [MAIN ] Completed service synchronization, ready to provide service. Jan 4 16:54:22 mds1 crmd: [1328]: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped! Jan 4 16:54:22 mds1 crmd: [1328]: WARN: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING Jan 4 16:54:22 mds1 crmd: [1328]: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ] Jan 4 16:54:22 mds1 crmd: [1328]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] Jan 4 16:54:22 mds1 crmd: [1328]: info: do_te_control: Registering TE UUID: 07e20dfe-052c-4b59-acd4-dd93542c2286 Jan 4 16:54:22 mds1 crmd: [1328]: WARN: cib_client_add_notify_callback: Callback already present Jan 4 16:54:22 mds1 crmd: [1328]: info: set_graph_functions: Setting custom graph functions Jan 4 16:54:22 mds1 crmd: [1328]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses Jan 4 16:54:22 mds1 crmd: [1328]: info: do_dc_takeover: Taking over DC status for this partition Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_readwrite: We are now in R/W mode Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/6, version=0.248.0): ok (rc=0) Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/7, version=0.248.0): ok (rc=0) Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/9, version=0.248.0): ok (rc=0) Jan 4 16:54:22 mds1 crmd: [1328]: info: join_make_offer: Making join offers based on membership 5080 Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/11, version=0.248.0): ok (rc=0) Jan 4 16:54:22 mds1 crmd: [1328]: info: do_dc_join_offer_all: join-1: Waiting on 4 outstanding join acks Jan 4 16:54:22 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5080: quorum retained Jan 4 16:54:22 mds1 crmd: [1328]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/14, version=0.248.0): ok (rc=0) Jan 4 16:54:22 mds1 crmd: [1328]: info: config_query_callback: Checking for expired actions every 900000ms Jan 4 16:54:22 mds1 crmd: [1328]: info: config_query_callback: Sending expected-votes=4 to corosync Jan 4 16:54:22 mds1 crmd: [1328]: info: update_dc: Set DC to mds1 (3.0.2) Jan 4 16:54:22 mds1 crmd: [1328]: info: ais_dispatch_message: Membership 5080: quorum retained Jan 4 16:54:22 mds1 crmd: [1328]: info: crmd_ais_dispatch: Setting expected votes to 4 Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/17, version=0.248.0): ok (rc=0) Jan 4 16:54:22 mds1 crmd: [1328]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 16:54:22 mds1 crmd: [1328]: info: do_state_transition: All 4 cluster nodes responded to the join offer. Jan 4 16:54:22 mds1 crmd: [1328]: info: do_dc_join_finalize: join-1: Syncing the CIB from mds2 to the rest of the cluster Jan 4 16:54:22 mds1 cib: [1324]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-83.raw Jan 4 16:54:22 mds1 cib: [1324]: info: write_cib_contents: Wrote version 0.249.0 of the CIB to disk (digest: 13bf82a1a274cb7401a804b8b2fcd551) Jan 4 16:54:22 mds1 cib: [1324]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.a5MdnD (digest: /var/lib/heartbeat/crm/cib.gfAErW) Jan 4 16:54:22 mds1 cib: [1324]: info: cib_replace_notify: Local-only Replace: -1.-1.-1 from mds2 Jan 4 16:54:22 mds1 cib: [1324]: info: log_data_element: cib:diff: - Jan 4 16:54:22 mds1 cib: [1324]: info: log_data_element: cib:diff: + Jan 4 16:54:22 mds1 attrd: [1326]: info: do_cib_replaced: Sending full refresh Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/19, version=0.249.0): ok (rc=0) Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/20, version=0.249.0): ok (rc=0) Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/21, version=0.249.0): ok (rc=0) Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/22, version=0.249.0): ok (rc=0) Jan 4 16:54:22 mds1 crmd: [1328]: info: do_dc_join_ack: join-1: Updating node state to member for oss1 Jan 4 16:54:22 mds1 crmd: [1328]: info: do_dc_join_ack: join-1: Updating node state to member for mds2 Jan 4 16:54:22 mds1 crmd: [1328]: info: do_dc_join_ack: join-1: Updating node state to member for oss2 Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/transient_attributes (origin=oss1/crmd/7, version=0.249.0): ok (rc=0) Jan 4 16:54:22 mds1 attrd: [1326]: info: crm_new_peer: Node oss1 now has id: 2474289344 Jan 4 16:54:22 mds1 crmd: [1328]: info: update_attrd: Connecting to attrd... Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds2']/transient_attributes (origin=mds2/crmd/7, version=0.249.0): ok (rc=0) Jan 4 16:54:22 mds1 attrd: [1326]: info: crm_new_peer: Node 2474289344 is now known as oss1 Jan 4 16:54:22 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss2']/transient_attributes (origin=oss2/crmd/7, version=0.249.0): ok (rc=0) Jan 4 16:54:22 mds1 crmd: [1328]: info: do_dc_join_ack: join-1: Updating node state to member for mds1 Jan 4 16:54:23 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for shutdown Jan 4 16:54:23 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss1']/lrm (origin=local/crmd/23, version=0.249.0): ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss1']/lrm": ok (rc=0) Jan 4 16:54:23 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds2']/lrm (origin=local/crmd/25, version=0.249.1): ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] Jan 4 16:54:23 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='oss2']/lrm (origin=local/crmd/27, version=0.249.2): ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 16:54:23 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds1']/transient_attributes (origin=local/crmd/29, version=0.249.3): ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date Jan 4 16:54:23 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='mds1']/lrm (origin=local/crmd/30, version=0.249.3): ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: crm_update_quorum: Updating quorum status to true (call=34) Jan 4 16:54:23 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/32, version=0.249.4): ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: abort_transition_graph: do_te_invoke:176 - Triggered transition abort (complete=1) : Peer Cancelled Jan 4 16:54:23 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for terminate Jan 4 16:54:23 mds1 crmd: [1328]: info: do_pe_invoke: Query 35: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:54:23 mds1 crmd: [1328]: info: erase_xpath_callback: Deletion of "//node_state[@uname='mds2']/lrm": ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: erase_xpath_callback: Deletion of "//node_state[@uname='oss2']/lrm": ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: erase_xpath_callback: Deletion of "//node_state[@uname='mds1']/transient_attributes": ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: erase_xpath_callback: Deletion of "//node_state[@uname='mds1']/lrm": ok (rc=0) Jan 4 16:54:23 mds1 cib: [1324]: info: write_cib_contents: Archived previous version as /var/lib/heartbeat/crm/cib-84.raw Jan 4 16:54:23 mds1 cib: [1324]: info: write_cib_contents: Wrote version 0.250.0 of the CIB to disk (digest: 6092bcd1a4f8fc1301c8ae67db63c7b8) Jan 4 16:54:23 mds1 cib: [1324]: info: retrieveCib: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.2Jsehg (digest: /var/lib/heartbeat/crm/cib.cxwO6z) Jan 4 16:54:23 mds1 cib: [1324]: info: log_data_element: cib:diff: - Jan 4 16:54:23 mds1 cib: [1324]: info: log_data_element: cib:diff: + Jan 4 16:54:23 mds1 cib: [1324]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/34, version=0.250.1): ok (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: abort_transition_graph: need_abort:59 - Triggered transition abort (complete=1) : Non-status change Jan 4 16:54:23 mds1 crmd: [1328]: info: need_abort: Aborting on change to admin_epoch Jan 4 16:54:23 mds1 crmd: [1328]: info: do_pe_invoke: Query 36: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:54:23 mds1 crmd: [1328]: info: do_pe_invoke_callback: Invoking the PE: query=36, ref=pe_calc-dc-1325714063-19, seq=5080, quorate=1 Jan 4 16:54:23 mds1 attrd: [1326]: info: crm_new_peer: Node mds2 now has id: 2608507072 Jan 4 16:54:23 mds1 attrd: [1326]: info: crm_new_peer: Node 2608507072 is now known as mds2 Jan 4 16:54:23 mds1 pengine: [1327]: info: unpack_config: Startup probes: enabled Jan 4 16:54:23 mds1 pengine: [1327]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 16:54:23 mds1 pengine: [1327]: info: unpack_domains: Unpacking domains Jan 4 16:54:23 mds1 attrd: [1326]: info: crm_new_peer: Node oss2 now has id: 2692393152 Jan 4 16:54:23 mds1 pengine: [1327]: info: determine_online_status: Node oss1 is online Jan 4 16:54:23 mds1 attrd: [1326]: info: crm_new_peer: Node 2692393152 is now known as oss2 Jan 4 16:54:23 mds1 pengine: [1327]: info: determine_online_status: Node mds2 is online Jan 4 16:54:23 mds1 attrd: [1326]: info: attrd_local_callback: Sending full refresh (origin=crmd) Jan 4 16:54:23 mds1 pengine: [1327]: info: determine_online_status: Node oss2 is online Jan 4 16:54:23 mds1 attrd: [1326]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown () Jan 4 16:54:23 mds1 pengine: [1327]: info: determine_online_status: Node mds1 is online Jan 4 16:54:23 mds1 attrd: [1326]: info: attrd_trigger_update: Sending flush op to all hosts for: terminate () Jan 4 16:54:23 mds1 pengine: [1327]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:23 mds1 pengine: [1327]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:23 mds1 pengine: [1327]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:23 mds1 pengine: [1327]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:23 mds1 pengine: [1327]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Stopped Jan 4 16:54:23 mds1 pengine: [1327]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Stopped Jan 4 16:54:23 mds1 pengine: [1327]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Stopped Jan 4 16:54:23 mds1 pengine: [1327]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Stopped Jan 4 16:54:23 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 16:54:23 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 16:54:23 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose mds2. Jan 4 16:54:23 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 16:54:23 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 16:54:23 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 16:54:23 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 16:54:23 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 16:54:23 mds1 pengine: [1327]: notice: LogActions: Start st-oss2#011(oss1) Jan 4 16:54:23 mds1 pengine: [1327]: notice: LogActions: Start st-oss1#011(oss2) Jan 4 16:54:23 mds1 pengine: [1327]: notice: LogActions: Start st-mds1#011(mds2) Jan 4 16:54:23 mds1 pengine: [1327]: notice: LogActions: Start st-mds2#011(mds1) Jan 4 16:54:23 mds1 pengine: [1327]: notice: LogActions: Start MGS_1#011(mds1) Jan 4 16:54:23 mds1 pengine: [1327]: notice: LogActions: Start lustre-MDT0000_2#011(mds1) Jan 4 16:54:23 mds1 pengine: [1327]: notice: LogActions: Start lustre-OST0001_4#011(oss1) Jan 4 16:54:23 mds1 pengine: [1327]: notice: LogActions: Start lustre-OST0000_3#011(oss1) Jan 4 16:54:23 mds1 crmd: [1328]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 16:54:23 mds1 crmd: [1328]: info: unpack_graph: Unpacked transition 0: 49 actions in 49 synapses Jan 4 16:54:23 mds1 crmd: [1328]: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1325714063-19) derived from /var/lib/pengine/pe-input-42.bz2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 4: monitor st-oss2_monitor_0 on oss1 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 13: monitor st-oss2_monitor_0 on oss2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 22: monitor st-oss2_monitor_0 on mds2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 31: monitor st-oss2_monitor_0 on mds1 (local) Jan 4 16:54:23 mds1 lrmd: [1325]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Jan 4 16:54:23 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=31:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286 op=st-oss2_monitor_0 ) Jan 4 16:54:23 mds1 lrmd: [1325]: info: rsc:st-oss2:2: probe Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 5: monitor st-oss1_monitor_0 on oss1 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 14: monitor st-oss1_monitor_0 on oss2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 23: monitor st-oss1_monitor_0 on mds2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 32: monitor st-oss1_monitor_0 on mds1 (local) Jan 4 16:54:23 mds1 lrmd: [1325]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Jan 4 16:54:23 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=32:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286 op=st-oss1_monitor_0 ) Jan 4 16:54:23 mds1 lrmd: [1325]: info: rsc:st-oss1:3: probe Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 6: monitor st-mds1_monitor_0 on oss1 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 15: monitor st-mds1_monitor_0 on oss2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 24: monitor st-mds1_monitor_0 on mds2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 33: monitor st-mds1_monitor_0 on mds1 (local) Jan 4 16:54:23 mds1 lrmd: [1325]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Jan 4 16:54:23 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=33:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286 op=st-mds1_monitor_0 ) Jan 4 16:54:23 mds1 lrmd: [1325]: info: rsc:st-mds1:4: probe Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 7: monitor st-mds2_monitor_0 on oss1 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 16: monitor st-mds2_monitor_0 on oss2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 25: monitor st-mds2_monitor_0 on mds2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 34: monitor st-mds2_monitor_0 on mds1 (local) Jan 4 16:54:23 mds1 lrmd: [1325]: notice: lrmd_rsc_new(): No lrm_rprovider field in message Jan 4 16:54:23 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=34:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286 op=st-mds2_monitor_0 ) Jan 4 16:54:23 mds1 lrmd: [1325]: info: rsc:st-mds2:5: probe Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 8: monitor MGS_1_monitor_0 on oss1 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 17: monitor MGS_1_monitor_0 on oss2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 26: monitor MGS_1_monitor_0 on mds2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 35: monitor MGS_1_monitor_0 on mds1 (local) Jan 4 16:54:23 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=35:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286 op=MGS_1_monitor_0 ) Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 9: monitor lustre-MDT0000_2_monitor_0 on oss1 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 18: monitor lustre-MDT0000_2_monitor_0 on oss2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 27: monitor lustre-MDT0000_2_monitor_0 on mds2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 36: monitor lustre-MDT0000_2_monitor_0 on mds1 (local) Jan 4 16:54:23 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=36:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286 op=lustre-MDT0000_2_monitor_0 ) Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 10: monitor lustre-OST0001_4_monitor_0 on oss1 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 19: monitor lustre-OST0001_4_monitor_0 on oss2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 28: monitor lustre-OST0001_4_monitor_0 on mds2 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 37: monitor lustre-OST0001_4_monitor_0 on mds1 (local) Jan 4 16:54:23 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=37:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286 op=lustre-OST0001_4_monitor_0 ) Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 11: monitor lustre-OST0000_3_monitor_0 on oss1 Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 20: monitor lustre-OST0000_3_monitor_0 on oss2 Jan 4 16:54:23 mds1 stonith-ng: [1323]: notice: stonith_device_action: Device st-oss1 not found Jan 4 16:54:23 mds1 stonith-ng: [1323]: info: stonith_command: Processed st_execute from lrmd: rc=-11 Jan 4 16:54:23 mds1 stonith-ng: [1323]: notice: stonith_device_action: Device st-mds1 not found Jan 4 16:54:23 mds1 stonith-ng: [1323]: info: stonith_command: Processed st_execute from lrmd: rc=-11 Jan 4 16:54:23 mds1 stonith-ng: [1323]: notice: stonith_device_action: Device st-oss2 not found Jan 4 16:54:23 mds1 stonith-ng: [1323]: info: stonith_command: Processed st_execute from lrmd: rc=-11 Jan 4 16:54:23 mds1 stonith-ng: [1323]: notice: stonith_device_action: Device st-mds2 not found Jan 4 16:54:23 mds1 stonith-ng: [1323]: info: stonith_command: Processed st_execute from lrmd: rc=-11 Jan 4 16:54:23 mds1 crmd: [1328]: info: process_lrm_event: LRM operation st-oss1_monitor_0 (call=3, rc=7, cib-update=37, confirmed=true) not running Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-oss1_monitor_0 (32) confirmed on mds1 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 29: monitor lustre-OST0000_3_monitor_0 on mds2 Jan 4 16:54:23 mds1 crmd: [1328]: info: process_lrm_event: LRM operation st-oss2_monitor_0 (call=2, rc=7, cib-update=38, confirmed=true) not running Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-oss2_monitor_0 (31) confirmed on mds1 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 38: monitor lustre-OST0000_3_monitor_0 on mds1 (local) Jan 4 16:54:23 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=38:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286 op=lustre-OST0000_3_monitor_0 ) Jan 4 16:54:23 mds1 lrmd: [1325]: info: rsc:lustre-OST0000_3:9: probe Jan 4 16:54:23 mds1 crmd: [1328]: info: process_lrm_event: LRM operation st-mds1_monitor_0 (call=4, rc=7, cib-update=39, confirmed=true) not running Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-mds1_monitor_0 (33) confirmed on mds1 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: process_lrm_event: LRM operation st-mds2_monitor_0 (call=5, rc=7, cib-update=40, confirmed=true) not running Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-mds2_monitor_0 (34) confirmed on mds1 (rc=0) Jan 4 16:54:23 mds1 pengine: [1327]: info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/pengine/pe-input-42.bz2 Jan 4 16:54:23 mds1 lrmd: [1325]: info: RA output: (lustre-OST0000_3:probe:stderr) Traceback (most recent call last):#012#012 File "/usr/bin/hydra-agent.py", line 144, in #012 result = args.func(args)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py", line 315, in target_running#012 info = store_get_target_info(args.uuid)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/store.py", line 26, in store_get_target_info#012 raise RuntimeError("Failed to read target data for '%s', is it configured? (%s)" % (name, e))#012#012RuntimeError: Failed to read target data for '1a698033-3abc-41e5-a278-d6f06e170348', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/1a698033-3abc-41e5-a278-d6f06e170348')#012 Jan 4 16:54:23 mds1 crmd: [1328]: info: process_lrm_event: LRM operation lustre-OST0000_3_monitor_0 (call=9, rc=0, cib-update=41, confirmed=true) ok Jan 4 16:54:23 mds1 crmd: [1328]: WARN: status_from_rc: Action 38 (lustre-OST0000_3_monitor_0) on mds1 failed (target: 7 vs. rc: 0): Error Jan 4 16:54:23 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0000_3_monitor_0, magic=0:0;38:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.6) : Event failed Jan 4 16:54:23 mds1 crmd: [1328]: info: update_abort_priority: Abort priority upgraded from 0 to 1 Jan 4 16:54:23 mds1 crmd: [1328]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (38) confirmed on mds1 (rc=4) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-oss2_monitor_0 (13) confirmed on oss2 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-oss1_monitor_0 (14) confirmed on oss2 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-mds2_monitor_0 (16) confirmed on oss2 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-mds1_monitor_0 (15) confirmed on oss2 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-mds1_monitor_0 (6) confirmed on oss1 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-oss2_monitor_0 (4) confirmed on oss1 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-oss1_monitor_0 (5) confirmed on oss1 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-mds2_monitor_0 (7) confirmed on oss1 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-oss2_monitor_0 (22) confirmed on mds2 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-oss1_monitor_0 (23) confirmed on mds2 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-mds2_monitor_0 (25) confirmed on mds2 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action st-mds1_monitor_0 (24) confirmed on mds2 (rc=0) Jan 4 16:54:23 mds1 crmd: [1328]: WARN: status_from_rc: Action 29 (lustre-OST0000_3_monitor_0) on mds2 failed (target: 7 vs. rc: 0): Error Jan 4 16:54:23 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0000_3_monitor_0, magic=0:0;29:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.19) : Event failed Jan 4 16:54:23 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (29) confirmed on mds2 (rc=4) Jan 4 16:54:24 mds1 lrmd: [1325]: info: rsc:MGS_1:6: probe Jan 4 16:54:24 mds1 lrmd: [1325]: info: rsc:lustre-MDT0000_2:7: probe Jan 4 16:54:24 mds1 lrmd: [1325]: info: rsc:lustre-OST0001_4:8: probe Jan 4 16:54:24 mds1 crmd: [1328]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (27) confirmed on mds2 (rc=0) Jan 4 16:54:24 mds1 crmd: [1328]: info: match_graph_event: Action MGS_1_monitor_0 (26) confirmed on mds2 (rc=0) Jan 4 16:54:24 mds1 crmd: [1328]: WARN: status_from_rc: Action 28 (lustre-OST0001_4_monitor_0) on mds2 failed (target: 7 vs. rc: 0): Error Jan 4 16:54:24 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0001_4_monitor_0, magic=0:0;28:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.22) : Event failed Jan 4 16:54:24 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (28) confirmed on mds2 (rc=4) Jan 4 16:54:24 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 21: probe_complete probe_complete on mds2 - no waiting Jan 4 16:54:24 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for probe_complete Jan 4 16:54:24 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds2, magic=NA, cib=0.250.23) : Transient attribute: update Jan 4 16:54:24 mds1 crmd: [1328]: info: update_abort_priority: Abort priority upgraded from 1 to 1000000 Jan 4 16:54:24 mds1 crmd: [1328]: info: update_abort_priority: 'Event failed' abort superceeded Jan 4 16:54:24 mds1 lrmd: [1325]: info: RA output: (lustre-OST0001_4:probe:stderr) Traceback (most recent call last):#012#012 File "/usr/bin/hydra-agent.py", line 144, in #012 result = args.func(args)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py", line 315, in target_running#012 info = store_get_target_info(args.uuid)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/store.py", line 26, in store_get_target_info#012 raise RuntimeError("Failed to read target data for '%s', is it configured? (%s)" % (name, e))#012#012RuntimeError: Failed to read target data for '00f07899-ff44-4d74-b5e7-e42cde69d607', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/00f07899-ff44-4d74-b5e7-e42cde69d607')#012 Jan 4 16:54:24 mds1 crmd: [1328]: info: process_lrm_event: LRM operation MGS_1_monitor_0 (call=6, rc=7, cib-update=42, confirmed=true) not running Jan 4 16:54:24 mds1 crmd: [1328]: info: match_graph_event: Action MGS_1_monitor_0 (35) confirmed on mds1 (rc=0) Jan 4 16:54:24 mds1 crmd: [1328]: info: process_lrm_event: LRM operation lustre-MDT0000_2_monitor_0 (call=7, rc=7, cib-update=43, confirmed=true) not running Jan 4 16:54:24 mds1 crmd: [1328]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (36) confirmed on mds1 (rc=0) Jan 4 16:54:24 mds1 crmd: [1328]: info: process_lrm_event: LRM operation lustre-OST0001_4_monitor_0 (call=8, rc=0, cib-update=44, confirmed=true) ok Jan 4 16:54:24 mds1 crmd: [1328]: WARN: status_from_rc: Action 37 (lustre-OST0001_4_monitor_0) on mds1 failed (target: 7 vs. rc: 0): Error Jan 4 16:54:24 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0001_4_monitor_0, magic=0:0;37:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.26) : Event failed Jan 4 16:54:24 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (37) confirmed on mds1 (rc=4) Jan 4 16:54:24 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 30: probe_complete probe_complete on mds1 (local) - no waiting Jan 4 16:54:24 mds1 attrd: [1326]: info: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true) Jan 4 16:54:24 mds1 attrd: [1326]: info: attrd_perform_update: Sent update 15: probe_complete=true Jan 4 16:54:24 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.250.27) : Transient attribute: update Jan 4 16:54:25 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (19) confirmed on oss2 (rc=0) Jan 4 16:54:25 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (20) confirmed on oss2 (rc=0) Jan 4 16:54:25 mds1 crmd: [1328]: WARN: status_from_rc: Action 18 (lustre-MDT0000_2_monitor_0) on oss2 failed (target: 7 vs. rc: 0): Error Jan 4 16:54:25 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_monitor_0, magic=0:0;18:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.30) : Event failed Jan 4 16:54:25 mds1 crmd: [1328]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (18) confirmed on oss2 (rc=4) Jan 4 16:54:25 mds1 crmd: [1328]: WARN: status_from_rc: Action 17 (MGS_1_monitor_0) on oss2 failed (target: 7 vs. rc: 0): Error Jan 4 16:54:25 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_monitor_0, magic=0:0;17:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.31) : Event failed Jan 4 16:54:25 mds1 crmd: [1328]: info: match_graph_event: Action MGS_1_monitor_0 (17) confirmed on oss2 (rc=4) Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 12: probe_complete probe_complete on oss2 - no waiting Jan 4 16:54:25 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss2, magic=NA, cib=0.250.32) : Transient attribute: update Jan 4 16:54:25 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0000_3_monitor_0 (11) confirmed on oss1 (rc=0) Jan 4 16:54:25 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0001_4_monitor_0 (10) confirmed on oss1 (rc=0) Jan 4 16:54:25 mds1 crmd: [1328]: WARN: status_from_rc: Action 8 (MGS_1_monitor_0) on oss1 failed (target: 7 vs. rc: 0): Error Jan 4 16:54:25 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_monitor_0, magic=0:0;8:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.35) : Event failed Jan 4 16:54:25 mds1 crmd: [1328]: info: match_graph_event: Action MGS_1_monitor_0 (8) confirmed on oss1 (rc=4) Jan 4 16:54:25 mds1 crmd: [1328]: WARN: status_from_rc: Action 9 (lustre-MDT0000_2_monitor_0) on oss1 failed (target: 7 vs. rc: 0): Error Jan 4 16:54:25 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_monitor_0, magic=0:0;9:0:7:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.36) : Event failed Jan 4 16:54:25 mds1 crmd: [1328]: info: match_graph_event: Action lustre-MDT0000_2_monitor_0 (9) confirmed on oss1 (rc=4) Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 3: probe_complete probe_complete on oss1 - no waiting Jan 4 16:54:25 mds1 crmd: [1328]: info: run_graph: ==================================================== Jan 4 16:54:25 mds1 crmd: [1328]: notice: run_graph: Transition 0 (Complete=36, Pending=0, Fired=0, Skipped=13, Incomplete=0, Source=/var/lib/pengine/pe-input-42.bz2): Stopped Jan 4 16:54:25 mds1 crmd: [1328]: info: te_graph_trigger: Transition 0 is now complete Jan 4 16:54:25 mds1 crmd: [1328]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 16:54:25 mds1 crmd: [1328]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 16:54:25 mds1 crmd: [1328]: info: do_pe_invoke: Query 45: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:54:25 mds1 crmd: [1328]: info: do_pe_invoke_callback: Invoking the PE: query=45, ref=pe_calc-dc-1325714065-56, seq=5080, quorate=1 Jan 4 16:54:25 mds1 pengine: [1327]: info: unpack_config: Startup probes: enabled Jan 4 16:54:25 mds1 pengine: [1327]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 16:54:25 mds1 pengine: [1327]: info: unpack_domains: Unpacking domains Jan 4 16:54:25 mds1 pengine: [1327]: info: determine_online_status: Node oss1 is online Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 16:54:25 mds1 pengine: [1327]: info: determine_online_status: Node mds2 is online Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 16:54:25 mds1 pengine: [1327]: info: determine_online_status: Node oss2 is online Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: native_add_running: Resource ocf::Target:lustre-MDT0000_2 appears to be active on 2 nodes. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: native_add_running: Resource ocf::Target:MGS_1 appears to be active on 2 nodes. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:54:25 mds1 pengine: [1327]: info: determine_online_status: Node mds1 is online Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: native_add_running: Resource ocf::Target:lustre-OST0000_3 appears to be active on 2 nodes. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: native_add_running: Resource ocf::Target:lustre-OST0001_4 appears to be active on 2 nodes. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: MGS_1#011(ocf::hydra:Target) Started Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0110 : oss1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0111 : oss2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target) Started Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0110 : oss1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0111 : oss2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target) Started Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0110 : mds2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0111 : mds1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target) Started Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0110 : mds2 Jan 4 16:54:25 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=1, tag=transient_attributes, id=oss1, magic=NA, cib=0.250.37) : Transient attribute: update Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0111 : mds1 Jan 4 16:54:25 mds1 crmd: [1328]: info: do_pe_invoke: Query 46: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:54:25 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 16:54:25 mds1 crmd: [1328]: info: do_pe_invoke_callback: Invoking the PE: query=46, ref=pe_calc-dc-1325714065-57, seq=5080, quorate=1 Jan 4 16:54:25 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 16:54:25 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose mds2. Jan 4 16:54:25 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: native_create_actions: Attempting recovery of resource MGS_1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 16:54:25 mds1 pengine: [1327]: WARN: native_create_actions: Attempting recovery of resource lustre-MDT0000_2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 16:54:25 mds1 pengine: [1327]: WARN: native_create_actions: Attempting recovery of resource lustre-OST0001_4 Jan 4 16:54:25 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 16:54:25 mds1 pengine: [1327]: WARN: native_create_actions: Attempting recovery of resource lustre-OST0000_3 Jan 4 16:54:25 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Start st-oss2#011(oss1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Start st-oss1#011(oss2) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Start st-mds1#011(mds2) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Start st-mds2#011(mds1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Move resource MGS_1#011(Started oss1 -> mds1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started oss1 -> mds1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-OST0001_4#011(Started mds2 -> oss1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-OST0000_3#011(Started mds2 -> oss1) Jan 4 16:54:25 mds1 crmd: [1328]: info: handle_response: pe_calc calculation pe_calc-dc-1325714065-56 is obsolete Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: process_pe_message: Transition 1: ERRORs found during PE processing. PEngine Input stored in: /var/lib/pengine/pe-error-14.bz2 Jan 4 16:54:25 mds1 pengine: [1327]: info: unpack_config: Startup probes: enabled Jan 4 16:54:25 mds1 pengine: [1327]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 16:54:25 mds1 pengine: [1327]: info: unpack_domains: Unpacking domains Jan 4 16:54:25 mds1 pengine: [1327]: info: determine_online_status: Node oss1 is online Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 16:54:25 mds1 pengine: [1327]: info: determine_online_status: Node mds2 is online Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 16:54:25 mds1 pengine: [1327]: info: determine_online_status: Node oss2 is online Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: native_add_running: Resource ocf::Target:lustre-MDT0000_2 appears to be active on 2 nodes. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: native_add_running: Resource ocf::Target:MGS_1 appears to be active on 2 nodes. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:54:25 mds1 pengine: [1327]: info: determine_online_status: Node mds1 is online Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: native_add_running: Resource ocf::Target:lustre-OST0000_3 appears to be active on 2 nodes. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:54:25 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 16:54:25 mds1 pengine: [1327]: ERROR: native_add_running: Resource ocf::Target:lustre-OST0001_4 appears to be active on 2 nodes. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: See http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more information. Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Stopped Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: MGS_1#011(ocf::hydra:Target) Started Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0110 : oss1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0111 : oss2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target) Started Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0110 : oss1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0111 : oss2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target) Started Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0110 : mds2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0111 : mds1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target) Started Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0110 : mds2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: native_print: #0111 : mds1 Jan 4 16:54:25 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 16:54:25 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 16:54:25 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose mds2. Jan 4 16:54:25 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose mds1. Jan 4 16:54:25 mds1 pengine: [1327]: WARN: native_create_actions: Attempting recovery of resource MGS_1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds1 Jan 4 16:54:25 mds1 pengine: [1327]: WARN: native_create_actions: Attempting recovery of resource lustre-MDT0000_2 Jan 4 16:54:25 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds1 Jan 4 16:54:25 mds1 pengine: [1327]: WARN: native_create_actions: Attempting recovery of resource lustre-OST0001_4 Jan 4 16:54:25 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 16:54:25 mds1 pengine: [1327]: WARN: native_create_actions: Attempting recovery of resource lustre-OST0000_3 Jan 4 16:54:25 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Start st-oss2#011(oss1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Start st-oss1#011(oss2) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Start st-mds1#011(mds2) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Start st-mds2#011(mds1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Move resource MGS_1#011(Started oss1 -> mds1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started oss1 -> mds1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-OST0001_4#011(Started mds2 -> oss1) Jan 4 16:54:25 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-OST0000_3#011(Started mds2 -> oss1) Jan 4 16:54:25 mds1 crmd: [1328]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 16:54:25 mds1 crmd: [1328]: info: unpack_graph: Unpacked transition 2: 21 actions in 21 synapses Jan 4 16:54:25 mds1 crmd: [1328]: info: do_te_invoke: Processing graph 2 (ref=pe_calc-dc-1325714065-57) derived from /var/lib/pengine/pe-error-15.bz2 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 7: start st-oss2_start_0 on oss1 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 8: start st-oss1_start_0 on oss2 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 9: start st-mds1_start_0 on mds2 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 10: start st-mds2_start_0 on mds1 (local) Jan 4 16:54:25 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=10:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286 op=st-mds2_start_0 ) Jan 4 16:54:25 mds1 lrmd: [1325]: info: rsc:st-mds2:10: start Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 11: stop MGS_1_stop_0 on oss1 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 12: stop MGS_1_stop_0 on oss2 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 15: stop lustre-MDT0000_2_stop_0 on oss1 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 16: stop lustre-MDT0000_2_stop_0 on oss2 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 19: stop lustre-OST0001_4_stop_0 on mds2 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 20: stop lustre-OST0001_4_stop_0 on mds1 (local) Jan 4 16:54:25 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=20:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286 op=lustre-OST0001_4_stop_0 ) Jan 4 16:54:25 mds1 lrmd: [1325]: info: rsc:lustre-OST0001_4:11: stop Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 23: stop lustre-OST0000_3_stop_0 on mds2 Jan 4 16:54:25 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 24: stop lustre-OST0000_3_stop_0 on mds1 (local) Jan 4 16:54:25 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=24:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286 op=lustre-OST0000_3_stop_0 ) Jan 4 16:54:25 mds1 lrmd: [1325]: info: rsc:lustre-OST0000_3:12: stop Jan 4 16:54:25 mds1 stonith-ng: [1323]: info: stonith_device_register: Added 'st-mds2' to the device list (1 active devices) Jan 4 16:54:25 mds1 stonith-ng: [1323]: info: stonith_command: Processed st_device_register from lrmd: rc=0 Jan 4 16:54:25 mds1 lrmd: [1325]: info: stonith_api_device_metadata: looking up fence_virsh/redhat metadata Jan 4 16:54:25 mds1 lrmd: [1325]: info: stonithRA plugin: got metadata: #012#012fence_virsh is an I/O Fencing agent which can be used with the virtual machines managed by libvirt. It logs via ssh to a dom0 and there run virsh command, which does all work. #012.P#012By default, virsh needs root account to do properly work. So you must allow ssh login in your sshd_config.#012#012#011#012#011#011#012#011#011#012#011#011Fencing Action#012#011#012#011#012#011#011#012#011#011#012#011#011IP Address or Hostname#012#011#012#011#012#011#011#012#011#011#012#011#011Login Name#012#011#012#011#012#011#011#012#011#011#012#011#011Login password or passphrase#012#011#012#011#012#011#011#012#011#011#012#011#011Script to retrieve password#012#011#012#011#012#011#011#012#011#011#012#011#011SSH connection#012#011#012#011#012#011#011#012 result = args.func(args)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py", line 230, in unmount_target#012 info = store_get_target_info(args.uuid)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/store.py", line 26, in store_get_target_info#012 raise RuntimeError("Failed to read target data for '%s', is it configured? (%s)" % (name, e))#012#012RuntimeError: Failed to read target data for '00f07899-ff44-4d74-b5e7-e42cde69d607', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/00f07899-ff44-4d74-b5e7-e42cde69d607')#012 Jan 4 16:54:26 mds1 lrmd: [1325]: info: RA output: (lustre-OST0000_3:stop:stderr) Traceback (most recent call last):#012#012 File "/usr/bin/hydra-agent.py", line 144, in #012 result = args.func(args)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py", line 230, in unmount_target#012 info = store_get_target_info(args.uuid)#012#012 File "/usr/lib/python2.6/site-packages/hydra_agent/store.py", line 26, in store_get_target_info#012 raise RuntimeError("Failed to read target data for '%s', is it configured? (%s)" % (name, e))#012#012RuntimeError: Failed to read target data for '1a698033-3abc-41e5-a278-d6f06e170348', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/1a698033-3abc-41e5-a278-d6f06e170348')#012 Jan 4 16:54:26 mds1 lrmd: [1325]: info: RA output: (lustre-OST0001_4:stop:stdout) {#012 "backtrace": "Traceback (most recent call last):\n\n File \"/usr/bin/hydra-agent.py\", line 144, in \n result = args.func(args)\n\n File \"/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py\", line 230, in unmount_target\n info = store_get_target_info(args.uuid)\n\n File \"/usr/lib/python2.6/site-packages/hydra_agent/store.py\", line 26, in store_get_target_info\n raise RuntimeError(\"Failed to read target data for '%s', is it configured? (%s)\" % (name, e))\n\nRuntimeError: Failed to read target data for '00f07899-ff44-4d74-b5e7-e42cde69d607', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/00f07899-ff44-4d74-b5e7-e42cde69d607')\n", #012 "exception": "cexceptions\nRuntimeError\np0\n(S\"Failed to read target data for '00f07899-ff44-4d74-b5e7-e42cde69d607', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/00f07899-ff44-4d74-b5e7-e42cde69d607')\"\np1\ntp2\nRp3\n.", #012 "success": false#012} Jan 4 16:54:26 mds1 crmd: [1328]: info: process_lrm_event: LRM operation lustre-OST0001_4_stop_0 (call=11, rc=0, cib-update=48, confirmed=true) ok Jan 4 16:54:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0001_4_stop_0 (20) confirmed on mds1 (rc=0) Jan 4 16:54:26 mds1 lrmd: [1325]: info: RA output: (lustre-OST0000_3:stop:stdout) {#012 "backtrace": "Traceback (most recent call last):\n\n File \"/usr/bin/hydra-agent.py\", line 144, in \n result = args.func(args)\n\n File \"/usr/lib/python2.6/site-packages/hydra_agent/actions/targets.py\", line 230, in unmount_target\n info = store_get_target_info(args.uuid)\n\n File \"/usr/lib/python2.6/site-packages/hydra_agent/store.py\", line 26, in store_get_target_info\n raise RuntimeError(\"Failed to read target data for '%s', is it configured? (%s)\" % (name, e))\n\nRuntimeError: Failed to read target data for '1a698033-3abc-41e5-a278-d6f06e170348', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/1a698033-3abc-41e5-a278-d6f06e170348')\n", #012 "exception": "cexceptions\nRuntimeError\np0\n(S\"Failed to read target data for '1a698033-3abc-41e5-a278-d6f06e170348', is it configured? ([Errno 2] No such file or directory: '/var/lib/hydra/1a698033-3abc-41e5-a278-d6f06e170348')\"\np1\ntp2\nRp3\n.", #012 "success": false#012} Jan 4 16:54:26 mds1 crmd: [1328]: info: process_lrm_event: LRM operation lustre-OST0000_3_stop_0 (call=12, rc=0, cib-update=49, confirmed=true) ok Jan 4 16:54:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0000_3_stop_0 (24) confirmed on mds1 (rc=0) Jan 4 16:54:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-MDT0000_2_stop_0 (16) confirmed on oss2 (rc=0) Jan 4 16:54:26 mds1 crmd: [1328]: info: match_graph_event: Action MGS_1_stop_0 (12) confirmed on oss2 (rc=0) Jan 4 16:54:26 mds1 crmd: [1328]: info: match_graph_event: Action MGS_1_stop_0 (11) confirmed on oss1 (rc=0) Jan 4 16:54:26 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 13: start MGS_1_start_0 on mds1 (local) Jan 4 16:54:26 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=13:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286 op=MGS_1_start_0 ) Jan 4 16:54:26 mds1 lrmd: [1325]: info: rsc:MGS_1:13: start Jan 4 16:54:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-MDT0000_2_stop_0 (15) confirmed on oss1 (rc=0) Jan 4 16:54:26 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 17: start lustre-MDT0000_2_start_0 on mds1 (local) Jan 4 16:54:26 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=17:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286 op=lustre-MDT0000_2_start_0 ) Jan 4 16:54:26 mds1 lrmd: [1325]: info: rsc:lustre-MDT0000_2:14: start Jan 4 16:54:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0001_4_stop_0 (19) confirmed on mds2 (rc=0) Jan 4 16:54:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0000_3_stop_0 (23) confirmed on mds2 (rc=0) Jan 4 16:54:26 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 21: start lustre-OST0001_4_start_0 on oss1 Jan 4 16:54:26 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 25: start lustre-OST0000_3_start_0 on oss1 Jan 4 16:54:26 mds1 crmd: [1328]: info: te_pseudo_action: Pseudo action 1 fired and confirmed Jan 4 16:54:27 mds1 kernel: Lustre: OBD class driver, http://wiki.whamcloud.com/ Jan 4 16:54:27 mds1 kernel: Lustre: Lustre Version: 2.1.0 Jan 4 16:54:27 mds1 kernel: Lustre: Build Version: RC2-g9d71fe8-PRISTINE-2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 Jan 4 16:54:27 mds1 kernel: Lustre: Lustre LU module (ffffffffa052bf60). Jan 4 16:54:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:54:57 mds1 kernel: fld: Unknown symbol RQF_FLD_QUERY Jan 4 16:55:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:55:27 mds1 kernel: fld: Unknown symbol req_capsule_server_pack Jan 4 16:55:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:55:57 mds1 kernel: fld: Unknown symbol req_capsule_client_get Jan 4 16:56:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:56:27 mds1 kernel: fld: Unknown symbol ptlrpc_queue_wait Jan 4 16:56:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:56:57 mds1 kernel: fld: Unknown symbol req_capsule_fini Jan 4 16:57:09 mds1 kernel: INFO: task modprobe:1650 blocked for more than 120 seconds. Jan 4 16:57:09 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 4 16:57:09 mds1 kernel: modprobe D 0000000000000000 0 1650 1642 0x00000080 Jan 4 16:57:09 mds1 kernel: ffff88002e8e9aa8 0000000000000086 0000000000000000 ffff88003e7260f8 Jan 4 16:57:09 mds1 kernel: ffff880002015f80 ffff88003e7260c0 ffff880002015f80 0000000000000000 Jan 4 16:57:09 mds1 kernel: ffff880037cf50b8 ffff88002e8e9fd8 000000000000f598 ffff880037cf50b8 Jan 4 16:57:09 mds1 kernel: Call Trace: Jan 4 16:57:09 mds1 kernel: [] schedule_timeout+0x215/0x2e0 Jan 4 16:57:09 mds1 kernel: [] ? autoremove_wake_function+0x16/0x40 Jan 4 16:57:09 mds1 kernel: [] ? __wake_up_common+0x59/0x90 Jan 4 16:57:09 mds1 kernel: [] wait_for_common+0x123/0x180 Jan 4 16:57:09 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 16:57:09 mds1 kernel: [] ? __queue_work+0x41/0x50 Jan 4 16:57:09 mds1 kernel: [] wait_for_completion+0x1d/0x20 Jan 4 16:57:09 mds1 kernel: [] call_usermodehelper_exec+0xe0/0xf0 Jan 4 16:57:09 mds1 kernel: [] ? lnet_startup_lndnis+0xe6/0x6e0 [lnet] Jan 4 16:57:09 mds1 kernel: [] __request_module+0x18b/0x210 Jan 4 16:57:09 mds1 kernel: [] ? lnet_parse_networks+0xb0/0x7e0 [lnet] Jan 4 16:57:09 mds1 kernel: [] ? pvclock_clocksource_read+0x58/0xd0 Jan 4 16:57:09 mds1 kernel: [] lnet_startup_lndnis+0xe6/0x6e0 [lnet] Jan 4 16:57:09 mds1 kernel: [] ? cfs_alloc+0x63/0x90 [libcfs] Jan 4 16:57:09 mds1 kernel: [] LNetNIInit+0x125/0x1f0 [lnet] Jan 4 16:57:09 mds1 kernel: [] ? __cond_resched+0x2a/0x40 Jan 4 16:57:09 mds1 kernel: [] ? init_module+0x0/0x597 [ptlrpc] Jan 4 16:57:09 mds1 kernel: [] ptlrpc_ni_init+0x29/0x170 [ptlrpc] Jan 4 16:57:09 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 16:57:09 mds1 kernel: [] ptlrpc_init_portals+0x13/0xd0 [ptlrpc] Jan 4 16:57:09 mds1 kernel: [] ? init_module+0x0/0x597 [ptlrpc] Jan 4 16:57:09 mds1 kernel: [] init_module+0xe0/0x597 [ptlrpc] Jan 4 16:57:09 mds1 kernel: [] ? __blocking_notifier_call_chain+0x65/0x80 Jan 4 16:57:09 mds1 kernel: [] do_one_initcall+0x3c/0x1d0 Jan 4 16:57:09 mds1 kernel: [] sys_init_module+0xdf/0x250 Jan 4 16:57:09 mds1 kernel: [] system_call_fastpath+0x16/0x1b Jan 4 16:57:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:57:27 mds1 kernel: fld: Unknown symbol req_capsule_init Jan 4 16:57:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:57:57 mds1 kernel: fld: Unknown symbol req_capsule_set Jan 4 16:58:27 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:58:27 mds1 kernel: fld: Unknown symbol req_capsule_server_get Jan 4 16:58:45 mds1 init: tty (/dev/tty1) main process (1599) killed by TERM signal Jan 4 16:58:45 mds1 init: tty (/dev/tty2) main process (1601) killed by TERM signal Jan 4 16:58:45 mds1 init: tty (/dev/tty3) main process (1603) killed by TERM signal Jan 4 16:58:45 mds1 init: tty (/dev/tty4) main process (1605) killed by TERM signal Jan 4 16:58:45 mds1 init: tty (/dev/tty5) main process (1607) killed by TERM signal Jan 4 16:58:45 mds1 init: tty (/dev/tty6) main process (1609) killed by TERM signal Jan 4 16:58:45 mds1 avahi-dnsconfd[1580]: Got SIGTERM, quitting. Jan 4 16:58:45 mds1 avahi-daemon[1355]: Got SIGTERM, quitting. Jan 4 16:58:45 mds1 avahi-daemon[1355]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.122.89. Jan 4 16:58:46 mds1 acpid: exiting Jan 4 16:58:46 mds1 corosync[1317]: [SERV ] Unloading all Corosync service engines. Jan 4 16:58:46 mds1 corosync[1317]: [pcmk ] notice: pcmk_shutdown: Shuting down Pacemaker Jan 4 16:58:46 mds1 corosync[1317]: [pcmk ] notice: stop_child: Sent -15 to crmd: [1328] Jan 4 16:58:46 mds1 crmd: [1328]: info: crm_signal_dispatch: Invoking handler for signal 15: Terminated Jan 4 16:58:46 mds1 crmd: [1328]: info: crm_shutdown: Requesting shutdown Jan 4 16:58:46 mds1 crmd: [1328]: WARN: do_log: FSA: Input I_SHUTDOWN from crm_shutdown() received in state S_TRANSITION_ENGINE Jan 4 16:58:46 mds1 crmd: [1328]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ] Jan 4 16:58:46 mds1 crmd: [1328]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 16:58:46 mds1 crmd: [1328]: info: do_shutdown_req: Sending shutdown request to DC: mds1 Jan 4 16:58:46 mds1 crmd: [1328]: info: handle_shutdown_request: Creating shutdown request for mds1 (state=S_POLICY_ENGINE) Jan 4 16:58:46 mds1 attrd: [1326]: info: attrd_trigger_update: Sending flush op to all hosts for: shutdown (1325714326) Jan 4 16:58:46 mds1 attrd: [1326]: info: attrd_perform_update: Sent update 22: shutdown=1325714326 Jan 4 16:58:46 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.250.51) : Transient attribute: update Jan 4 16:58:46 mds1 crmd: [1328]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Jan 4 16:58:46 mds1 crmd: [1328]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 16:58:57 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:58:57 mds1 kernel: fld: Unknown symbol ptlrpc_at_set_req_timeout Jan 4 16:59:09 mds1 kernel: INFO: task modprobe:1650 blocked for more than 120 seconds. Jan 4 16:59:09 mds1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Jan 4 16:59:09 mds1 kernel: modprobe D 0000000000000000 0 1650 1642 0x00000080 Jan 4 16:59:09 mds1 kernel: ffff88002e8e9aa8 0000000000000086 0000000000000000 ffff88003e7260f8 Jan 4 16:59:09 mds1 kernel: ffff880002015f80 ffff88003e7260c0 ffff880002015f80 0000000000000000 Jan 4 16:59:09 mds1 kernel: ffff880037cf50b8 ffff88002e8e9fd8 000000000000f598 ffff880037cf50b8 Jan 4 16:59:09 mds1 kernel: Call Trace: Jan 4 16:59:09 mds1 kernel: [] schedule_timeout+0x215/0x2e0 Jan 4 16:59:09 mds1 kernel: [] ? autoremove_wake_function+0x16/0x40 Jan 4 16:59:09 mds1 kernel: [] ? __wake_up_common+0x59/0x90 Jan 4 16:59:09 mds1 kernel: [] wait_for_common+0x123/0x180 Jan 4 16:59:09 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 16:59:09 mds1 kernel: [] ? __queue_work+0x41/0x50 Jan 4 16:59:09 mds1 kernel: [] wait_for_completion+0x1d/0x20 Jan 4 16:59:09 mds1 kernel: [] call_usermodehelper_exec+0xe0/0xf0 Jan 4 16:59:09 mds1 kernel: [] ? lnet_startup_lndnis+0xe6/0x6e0 [lnet] Jan 4 16:59:09 mds1 kernel: [] __request_module+0x18b/0x210 Jan 4 16:59:09 mds1 kernel: [] ? lnet_parse_networks+0xb0/0x7e0 [lnet] Jan 4 16:59:09 mds1 kernel: [] ? pvclock_clocksource_read+0x58/0xd0 Jan 4 16:59:09 mds1 kernel: [] lnet_startup_lndnis+0xe6/0x6e0 [lnet] Jan 4 16:59:09 mds1 kernel: [] ? cfs_alloc+0x63/0x90 [libcfs] Jan 4 16:59:09 mds1 kernel: [] LNetNIInit+0x125/0x1f0 [lnet] Jan 4 16:59:09 mds1 kernel: [] ? __cond_resched+0x2a/0x40 Jan 4 16:59:09 mds1 kernel: [] ? init_module+0x0/0x597 [ptlrpc] Jan 4 16:59:09 mds1 kernel: [] ptlrpc_ni_init+0x29/0x170 [ptlrpc] Jan 4 16:59:09 mds1 kernel: [] ? default_wake_function+0x0/0x20 Jan 4 16:59:09 mds1 kernel: [] ptlrpc_init_portals+0x13/0xd0 [ptlrpc] Jan 4 16:59:09 mds1 kernel: [] ? init_module+0x0/0x597 [ptlrpc] Jan 4 16:59:09 mds1 kernel: [] init_module+0xe0/0x597 [ptlrpc] Jan 4 16:59:09 mds1 kernel: [] ? __blocking_notifier_call_chain+0x65/0x80 Jan 4 16:59:09 mds1 kernel: [] do_one_initcall+0x3c/0x1d0 Jan 4 16:59:09 mds1 kernel: [] sys_init_module+0xdf/0x250 Jan 4 16:59:09 mds1 kernel: [] system_call_fastpath+0x16/0x1b Jan 4 16:59:16 mds1 corosync[1317]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1328, seq=6) to terminate... Jan 4 16:59:26 mds1 lrmd: [1325]: WARN: MGS_1:start process (PID 1642) timed out (try 1). Killing with signal SIGTERM (15). Jan 4 16:59:26 mds1 lrmd: [1325]: WARN: operation start[13] on ocf::Target::MGS_1 for client 1328, its parameters: CRM_meta_name=[start] crm_feature_set=[3.0.2] target=[96c8dccb-368b-4ffc-af7e-6b95cdcacadb] CRM_meta_timeout=[300000] : pid [1642] timed out Jan 4 16:59:26 mds1 crmd: [1328]: ERROR: process_lrm_event: LRM operation MGS_1_start_0 (13) Timed Out (timeout=300000ms) Jan 4 16:59:26 mds1 crmd: [1328]: WARN: status_from_rc: Action 13 (MGS_1_start_0) on mds1 failed (target: 0 vs. rc: -2): Error Jan 4 16:59:26 mds1 crmd: [1328]: WARN: update_failcount: Updating failcount for MGS_1 on mds1 after failed start: rc=-2 (update=INFINITY, time=1325714366) Jan 4 16:59:26 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=MGS_1_start_0, magic=2:-2;13:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.52) : Event failed Jan 4 16:59:26 mds1 crmd: [1328]: info: match_graph_event: Action MGS_1_start_0 (13) confirmed on mds1 (rc=4) Jan 4 16:59:26 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for fail-count-MGS_1 Jan 4 16:59:26 mds1 attrd: [1326]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-MGS_1 (INFINITY) Jan 4 16:59:26 mds1 attrd: [1326]: info: attrd_perform_update: Sent update 25: fail-count-MGS_1=INFINITY Jan 4 16:59:26 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.250.53) : Transient attribute: update Jan 4 16:59:26 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for last-failure-MGS_1 Jan 4 16:59:26 mds1 attrd: [1326]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-MGS_1 (1325714366) Jan 4 16:59:26 mds1 attrd: [1326]: info: attrd_perform_update: Sent update 28: last-failure-MGS_1=1325714366 Jan 4 16:59:26 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.250.54) : Transient attribute: update Jan 4 16:59:26 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:59:26 mds1 kernel: fld: Unknown symbol ptlrpc_request_alloc_pack Jan 4 16:59:26 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:59:26 mds1 kernel: fld: Unknown symbol RMF_FLD_OPC Jan 4 16:59:26 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:59:26 mds1 kernel: fld: Unknown symbol ptlrpc_request_set_replen Jan 4 16:59:26 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:59:26 mds1 kernel: fld: Unknown symbol RMF_FLD_MDFLD Jan 4 16:59:26 mds1 kernel: fld: gave up waiting for init of module ptlrpc. Jan 4 16:59:26 mds1 kernel: fld: Unknown symbol ptlrpc_req_finished Jan 4 16:59:26 mds1 lrmd: [1325]: WARN: lustre-MDT0000_2:start process (PID 1646) timed out (try 1). Killing with signal SIGTERM (15). Jan 4 16:59:26 mds1 lrmd: [1325]: WARN: operation start[14] on ocf::Target::lustre-MDT0000_2 for client 1328, its parameters: CRM_meta_name=[start] crm_feature_set=[3.0.2] target=[c4f08837-0cea-46c1-a5d8-4ef412fadd0b] CRM_meta_timeout=[300000] : pid [1646] timed out Jan 4 16:59:26 mds1 crmd: [1328]: ERROR: process_lrm_event: LRM operation lustre-MDT0000_2_start_0 (14) Timed Out (timeout=300000ms) Jan 4 16:59:26 mds1 crmd: [1328]: WARN: status_from_rc: Action 17 (lustre-MDT0000_2_start_0) on mds1 failed (target: 0 vs. rc: -2): Error Jan 4 16:59:26 mds1 crmd: [1328]: WARN: update_failcount: Updating failcount for lustre-MDT0000_2 on mds1 after failed start: rc=-2 (update=INFINITY, time=1325714366) Jan 4 16:59:26 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-MDT0000_2_start_0, magic=2:-2;17:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.55) : Event failed Jan 4 16:59:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-MDT0000_2_start_0 (17) confirmed on mds1 (rc=4) Jan 4 16:59:26 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for fail-count-lustre-MDT0000_2 Jan 4 16:59:26 mds1 attrd: [1326]: info: attrd_trigger_update: Sending flush op to all hosts for: fail-count-lustre-MDT0000_2 (INFINITY) Jan 4 16:59:26 mds1 attrd: [1326]: info: attrd_perform_update: Sent update 31: fail-count-lustre-MDT0000_2=INFINITY Jan 4 16:59:26 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.250.56) : Transient attribute: update Jan 4 16:59:26 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for last-failure-lustre-MDT0000_2 Jan 4 16:59:26 mds1 attrd: [1326]: info: attrd_trigger_update: Sending flush op to all hosts for: last-failure-lustre-MDT0000_2 (1325714366) Jan 4 16:59:26 mds1 attrd: [1326]: info: attrd_perform_update: Sent update 34: last-failure-lustre-MDT0000_2=1325714366 Jan 4 16:59:26 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=mds1, magic=NA, cib=0.250.57) : Transient attribute: update Jan 4 16:59:26 mds1 kernel: LustreError: 1650:0:(socklnd.c:2420:ksocknal_base_startup()) Can't spawn socknal scheduler[0]: -513 Jan 4 16:59:26 mds1 kernel: LustreError: 105-4: Error -100 starting up LNI tcp Jan 4 16:59:26 mds1 kernel: LustreError: 1650:0:(events.c:728:ptlrpc_init_portals()) network initialisation failed Jan 4 16:59:26 mds1 crmd: [1328]: WARN: status_from_rc: Action 21 (lustre-OST0001_4_start_0) on oss1 failed (target: 0 vs. rc: -2): Error Jan 4 16:59:26 mds1 crmd: [1328]: WARN: update_failcount: Updating failcount for lustre-OST0001_4 on oss1 after failed start: rc=-2 (update=INFINITY, time=1325714366) Jan 4 16:59:26 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0001_4_start_0, magic=2:-2;21:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.58) : Event failed Jan 4 16:59:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0001_4_start_0 (21) confirmed on oss1 (rc=4) Jan 4 16:59:26 mds1 crmd: [1328]: WARN: status_from_rc: Action 25 (lustre-OST0000_3_start_0) on oss1 failed (target: 0 vs. rc: -2): Error Jan 4 16:59:26 mds1 crmd: [1328]: WARN: update_failcount: Updating failcount for lustre-OST0000_3 on oss1 after failed start: rc=-2 (update=INFINITY, time=1325714366) Jan 4 16:59:26 mds1 crmd: [1328]: info: abort_transition_graph: match_graph_event:276 - Triggered transition abort (complete=0, tag=lrm_rsc_op, id=lustre-OST0000_3_start_0, magic=2:-2;25:2:0:07e20dfe-052c-4b59-acd4-dd93542c2286, cib=0.250.59) : Event failed Jan 4 16:59:26 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0000_3_start_0 (25) confirmed on oss1 (rc=4) Jan 4 16:59:26 mds1 crmd: [1328]: info: run_graph: ==================================================== Jan 4 16:59:26 mds1 crmd: [1328]: notice: run_graph: Transition 2 (Complete=17, Pending=0, Fired=0, Skipped=4, Incomplete=0, Source=/var/lib/pengine/pe-error-15.bz2): Stopped Jan 4 16:59:26 mds1 crmd: [1328]: info: te_graph_trigger: Transition 2 is now complete Jan 4 16:59:26 mds1 crmd: [1328]: info: do_pe_invoke: Query 53: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:59:26 mds1 crmd: [1328]: info: do_pe_invoke_callback: Invoking the PE: query=53, ref=pe_calc-dc-1325714366-75, seq=5080, quorate=1 Jan 4 16:59:26 mds1 pengine: [1327]: info: unpack_config: Startup probes: enabled Jan 4 16:59:26 mds1 pengine: [1327]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 16:59:26 mds1 pengine: [1327]: info: unpack_domains: Unpacking domains Jan 4 16:59:26 mds1 pengine: [1327]: info: determine_online_status: Node oss1 is online Jan 4 16:59:26 mds1 pengine: [1327]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 16:59:26 mds1 pengine: [1327]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 16:59:26 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 16:59:26 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 16:59:26 mds1 pengine: [1327]: info: determine_online_status: Node mds2 is online Jan 4 16:59:26 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 16:59:26 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 16:59:26 mds1 pengine: [1327]: info: determine_online_status: Node oss2 is online Jan 4 16:59:26 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 16:59:26 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 16:59:26 mds1 pengine: [1327]: info: determine_online_status: Node mds1 is shutting down Jan 4 16:59:26 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 16:59:26 mds1 pengine: [1327]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 16:59:26 mds1 pengine: [1327]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 16:59:26 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 16:59:26 mds1 pengine: [1327]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started oss1 Jan 4 16:59:26 mds1 pengine: [1327]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started oss2 Jan 4 16:59:26 mds1 pengine: [1327]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds2 Jan 4 16:59:26 mds1 pengine: [1327]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 16:59:26 mds1 pengine: [1327]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 16:59:26 mds1 pengine: [1327]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 16:59:26 mds1 pengine: [1327]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 16:59:26 mds1 pengine: [1327]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 16:59:26 mds1 pengine: [1327]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 16:59:26 mds1 pengine: [1327]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 16:59:26 mds1 pengine: [1327]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 16:59:26 mds1 pengine: [1327]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 16:59:26 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 16:59:26 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 16:59:26 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose mds2. Jan 4 16:59:26 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose oss1. Jan 4 16:59:26 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 16:59:26 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds2 Jan 4 16:59:27 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss1 Jan 4 16:59:27 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss1 Jan 4 16:59:27 mds1 pengine: [1327]: info: stage6: Scheduling Node mds1 for shutdown Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Leave resource st-oss2#011(Started oss1) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Leave resource st-oss1#011(Started oss2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Leave resource st-mds1#011(Started mds2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Move resource st-mds2#011(Started mds1 -> oss1) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds1 -> mds2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Recover resource lustre-OST0001_4#011(Started oss1) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Recover resource lustre-OST0000_3#011(Started oss1) Jan 4 16:59:27 mds1 crmd: [1328]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 16:59:27 mds1 crmd: [1328]: info: unpack_graph: Unpacked transition 3: 16 actions in 16 synapses Jan 4 16:59:27 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for fail-count-lustre-OST0001_4 Jan 4 16:59:27 mds1 crmd: [1328]: info: do_te_invoke: Processing graph 3 (ref=pe_calc-dc-1325714366-75) derived from /var/lib/pengine/pe-input-43.bz2 Jan 4 16:59:27 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for last-failure-lustre-OST0001_4 Jan 4 16:59:27 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.250.60) : Transient attribute: update Jan 4 16:59:27 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for fail-count-lustre-OST0000_3 Jan 4 16:59:27 mds1 crmd: [1328]: info: update_abort_priority: Abort priority upgraded from 0 to 1000000 Jan 4 16:59:27 mds1 attrd: [1326]: info: find_hash_entry: Creating hash entry for last-failure-lustre-OST0000_3 Jan 4 16:59:27 mds1 crmd: [1328]: info: update_abort_priority: Abort action done superceeded by restart Jan 4 16:59:27 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.250.61) : Transient attribute: update Jan 4 16:59:27 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.250.62) : Transient attribute: update Jan 4 16:59:27 mds1 crmd: [1328]: info: abort_transition_graph: te_update_diff:146 - Triggered transition abort (complete=0, tag=transient_attributes, id=oss1, magic=NA, cib=0.250.63) : Transient attribute: update Jan 4 16:59:27 mds1 crmd: [1328]: info: run_graph: ==================================================== Jan 4 16:59:27 mds1 crmd: [1328]: notice: run_graph: Transition 3 (Complete=0, Pending=0, Fired=0, Skipped=16, Incomplete=0, Source=/var/lib/pengine/pe-input-43.bz2): Stopped Jan 4 16:59:27 mds1 crmd: [1328]: info: te_graph_trigger: Transition 3 is now complete Jan 4 16:59:27 mds1 crmd: [1328]: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ] Jan 4 16:59:27 mds1 crmd: [1328]: info: do_state_transition: All 4 cluster nodes are eligible to run resources. Jan 4 16:59:27 mds1 crmd: [1328]: info: do_pe_invoke: Query 54: Requesting the current CIB: S_POLICY_ENGINE Jan 4 16:59:27 mds1 crmd: [1328]: info: do_pe_invoke_callback: Invoking the PE: query=54, ref=pe_calc-dc-1325714367-76, seq=5080, quorate=1 Jan 4 16:59:27 mds1 pengine: [1327]: info: process_pe_message: Transition 3: PEngine Input stored in: /var/lib/pengine/pe-input-43.bz2 Jan 4 16:59:27 mds1 pengine: [1327]: info: unpack_config: Startup probes: enabled Jan 4 16:59:27 mds1 pengine: [1327]: info: unpack_config: Node scores: 'red' = -INFINITY, 'yellow' = 0, 'green' = 0 Jan 4 16:59:27 mds1 pengine: [1327]: info: unpack_domains: Unpacking domains Jan 4 16:59:27 mds1 pengine: [1327]: info: determine_online_status: Node oss1 is online Jan 4 16:59:27 mds1 pengine: [1327]: WARN: unpack_rsc_op: Processing failed op lustre-OST0000_3_start_0 on oss1: unknown exec error (-2) Jan 4 16:59:27 mds1 pengine: [1327]: WARN: unpack_rsc_op: Processing failed op lustre-OST0001_4_start_0 on oss1: unknown exec error (-2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss1 Jan 4 16:59:27 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss1 Jan 4 16:59:27 mds1 pengine: [1327]: info: determine_online_status: Node mds2 is online Jan 4 16:59:27 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds2 Jan 4 16:59:27 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds2 Jan 4 16:59:27 mds1 pengine: [1327]: info: determine_online_status: Node oss2 is online Jan 4 16:59:27 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-MDT0000_2_monitor_0 found resource lustre-MDT0000_2 active on oss2 Jan 4 16:59:27 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation MGS_1_monitor_0 found resource MGS_1 active on oss2 Jan 4 16:59:27 mds1 pengine: [1327]: info: determine_online_status: Node mds1 is shutting down Jan 4 16:59:27 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0000_3_monitor_0 found resource lustre-OST0000_3 active on mds1 Jan 4 16:59:27 mds1 pengine: [1327]: WARN: unpack_rsc_op: Processing failed op MGS_1_start_0 on mds1: unknown exec error (-2) Jan 4 16:59:27 mds1 pengine: [1327]: WARN: unpack_rsc_op: Processing failed op lustre-MDT0000_2_start_0 on mds1: unknown exec error (-2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: unpack_rsc_op: Operation lustre-OST0001_4_monitor_0 found resource lustre-OST0001_4 active on mds1 Jan 4 16:59:27 mds1 pengine: [1327]: notice: native_print: st-oss2#011(stonith:fence_virsh):#011Started oss1 Jan 4 16:59:27 mds1 pengine: [1327]: notice: native_print: st-oss1#011(stonith:fence_virsh):#011Started oss2 Jan 4 16:59:27 mds1 pengine: [1327]: notice: native_print: st-mds1#011(stonith:fence_virsh):#011Started mds2 Jan 4 16:59:27 mds1 pengine: [1327]: notice: native_print: st-mds2#011(stonith:fence_virsh):#011Started mds1 Jan 4 16:59:27 mds1 pengine: [1327]: notice: native_print: MGS_1#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 16:59:27 mds1 pengine: [1327]: notice: native_print: lustre-MDT0000_2#011(ocf::hydra:Target):#011Started mds1 FAILED Jan 4 16:59:27 mds1 pengine: [1327]: notice: native_print: lustre-OST0001_4#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 16:59:27 mds1 pengine: [1327]: notice: native_print: lustre-OST0000_3#011(ocf::hydra:Target):#011Started oss1 FAILED Jan 4 16:59:27 mds1 pengine: [1327]: info: get_failcount: lustre-OST0001_4 has failed INFINITY times on oss1 Jan 4 16:59:27 mds1 pengine: [1327]: WARN: common_apply_stickiness: Forcing lustre-OST0001_4 away from oss1 after 1000000 failures (max=1000000) Jan 4 16:59:27 mds1 pengine: [1327]: info: get_failcount: lustre-OST0000_3 has failed INFINITY times on oss1 Jan 4 16:59:27 mds1 pengine: [1327]: WARN: common_apply_stickiness: Forcing lustre-OST0000_3 away from oss1 after 1000000 failures (max=1000000) Jan 4 16:59:27 mds1 pengine: [1327]: info: get_failcount: MGS_1 has failed INFINITY times on mds1 Jan 4 16:59:27 mds1 pengine: [1327]: WARN: common_apply_stickiness: Forcing MGS_1 away from mds1 after 1000000 failures (max=1000000) Jan 4 16:59:27 mds1 pengine: [1327]: info: get_failcount: lustre-MDT0000_2 has failed INFINITY times on mds1 Jan 4 16:59:27 mds1 pengine: [1327]: WARN: common_apply_stickiness: Forcing lustre-MDT0000_2 away from mds1 after 1000000 failures (max=1000000) Jan 4 16:59:27 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss2 resources. Chose oss1. Jan 4 16:59:27 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-oss1 resources. Chose oss2. Jan 4 16:59:27 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds1 resources. Chose mds2. Jan 4 16:59:27 mds1 pengine: [1327]: info: native_choose_node: 2 nodes with equal score (100) for running st-mds2 resources. Chose oss1. Jan 4 16:59:27 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for MGS_1 on mds2 Jan 4 16:59:27 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-MDT0000_2 on mds2 Jan 4 16:59:27 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0001_4 on oss2 Jan 4 16:59:27 mds1 pengine: [1327]: notice: RecurringOp: Start recurring monitor (120s) for lustre-OST0000_3 on oss2 Jan 4 16:59:27 mds1 pengine: [1327]: info: stage6: Scheduling Node mds1 for shutdown Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Leave resource st-oss2#011(Started oss1) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Leave resource st-oss1#011(Started oss2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Leave resource st-mds1#011(Started mds2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Move resource st-mds2#011(Started mds1 -> oss1) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Move resource MGS_1#011(Started mds1 -> mds2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-MDT0000_2#011(Started mds1 -> mds2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-OST0001_4#011(Started oss1 -> oss2) Jan 4 16:59:27 mds1 pengine: [1327]: notice: LogActions: Move resource lustre-OST0000_3#011(Started oss1 -> oss2) Jan 4 16:59:27 mds1 crmd: [1328]: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Jan 4 16:59:27 mds1 crmd: [1328]: info: unpack_graph: Unpacked transition 4: 16 actions in 16 synapses Jan 4 16:59:27 mds1 crmd: [1328]: info: do_te_invoke: Processing graph 4 (ref=pe_calc-dc-1325714367-76) derived from /var/lib/pengine/pe-input-44.bz2 Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 17: stop st-mds2_stop_0 on mds1 (local) Jan 4 16:59:27 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=17:4:0:07e20dfe-052c-4b59-acd4-dd93542c2286 op=st-mds2_stop_0 ) Jan 4 16:59:27 mds1 lrmd: [1325]: info: rsc:st-mds2:15: stop Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 3: stop MGS_1_stop_0 on mds1 (local) Jan 4 16:59:27 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=3:4:0:07e20dfe-052c-4b59-acd4-dd93542c2286 op=MGS_1_stop_0 ) Jan 4 16:59:27 mds1 lrmd: [1325]: info: rsc:MGS_1:16: stop Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 4: stop lustre-MDT0000_2_stop_0 on mds1 (local) Jan 4 16:59:27 mds1 crmd: [1328]: info: do_lrm_rsc_op: Performing key=4:4:0:07e20dfe-052c-4b59-acd4-dd93542c2286 op=lustre-MDT0000_2_stop_0 ) Jan 4 16:59:27 mds1 lrmd: [1325]: info: rsc:lustre-MDT0000_2:17: stop Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 2: stop lustre-OST0001_4_stop_0 on oss1 Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 1: stop lustre-OST0000_3_stop_0 on oss1 Jan 4 16:59:27 mds1 stonith-ng: [1323]: info: stonith_device_remove: Removed 'st-mds2' from the device list (0 active devices) Jan 4 16:59:27 mds1 stonith-ng: [1323]: info: stonith_command: Processed st_device_remove from lrmd: rc=0 Jan 4 16:59:27 mds1 crmd: [1328]: info: process_lrm_event: LRM operation st-mds2_stop_0 (call=15, rc=0, cib-update=55, confirmed=true) ok Jan 4 16:59:27 mds1 crmd: [1328]: info: match_graph_event: Action st-mds2_stop_0 (17) confirmed on mds1 (rc=0) Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 18: start st-mds2_start_0 on oss1 Jan 4 16:59:27 mds1 pengine: [1327]: info: process_pe_message: Transition 4: PEngine Input stored in: /var/lib/pengine/pe-input-44.bz2 Jan 4 16:59:27 mds1 crmd: [1328]: info: match_graph_event: Action st-mds2_start_0 (18) confirmed on oss1 (rc=0) Jan 4 16:59:27 mds1 crmd: [1328]: info: process_lrm_event: LRM operation MGS_1_stop_0 (call=16, rc=0, cib-update=56, confirmed=true) ok Jan 4 16:59:27 mds1 crmd: [1328]: info: match_graph_event: Action MGS_1_stop_0 (3) confirmed on mds1 (rc=0) Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 19: start MGS_1_start_0 on mds2 Jan 4 16:59:27 mds1 crmd: [1328]: info: process_lrm_event: LRM operation lustre-MDT0000_2_stop_0 (call=17, rc=0, cib-update=57, confirmed=true) ok Jan 4 16:59:27 mds1 crmd: [1328]: info: match_graph_event: Action lustre-MDT0000_2_stop_0 (4) confirmed on mds1 (rc=0) Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 21: start lustre-MDT0000_2_start_0 on mds2 Jan 4 16:59:27 mds1 crmd: [1328]: info: te_crm_command: Executing crm-event (29): do_shutdown on mds1 Jan 4 16:59:27 mds1 crmd: [1328]: info: te_crm_command: crm-event (29) is a local shutdown Jan 4 16:59:27 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0000_3_stop_0 (1) confirmed on oss1 (rc=0) Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 25: start lustre-OST0000_3_start_0 on oss2 Jan 4 16:59:27 mds1 crmd: [1328]: info: match_graph_event: Action lustre-OST0001_4_stop_0 (2) confirmed on oss1 (rc=0) Jan 4 16:59:27 mds1 crmd: [1328]: info: te_rsc_command: Initiating action 23: start lustre-OST0001_4_start_0 on oss2 Jan 4 16:59:27 mds1 crmd: [1328]: info: te_pseudo_action: Pseudo action 5 fired and confirmed Jan 4 16:59:46 mds1 corosync[1317]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1328, seq=6) to terminate... Jan 4 17:00:16 mds1 corosync[1317]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1328, seq=6) to terminate... Jan 4 17:00:46 mds1 corosync[1317]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1328, seq=6) to terminate... Jan 4 17:01:16 mds1 corosync[1317]: [pcmk ] notice: pcmk_shutdown: Still waiting for crmd (pid=1328, seq=6) to terminate... Jan 4 17:02:10 mds1 kernel: imklog 4.6.2, log source = /proc/kmsg started. Jan 4 17:02:10 mds1 rsyslogd: [origin software="rsyslogd" swVersion="4.6.2" x-pid="1166" x-info="http://www.rsyslog.com"] (re)start Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys cpuset Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys cpu Jan 4 17:02:10 mds1 kernel: Linux version 2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 (jenkins@rhel6-64-build01.lab.whamcloud.com) (gcc version 4.4.4 20100726 (Red Hat 4.4.4-13) (GCC) ) #1 SMP Sat Aug 20 09:56:49 PDT 2011 Jan 4 17:02:10 mds1 kernel: Command line: ro root=/dev/mapper/LustreVG-root rd_LVM_LV=LustreVG/root rd_LVM_LV=LustreVG/swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=auto rhgb quiet Jan 4 17:02:10 mds1 kernel: KERNEL supported cpus: Jan 4 17:02:10 mds1 kernel: Intel GenuineIntel Jan 4 17:02:10 mds1 kernel: AMD AuthenticAMD Jan 4 17:02:10 mds1 kernel: Centaur CentaurHauls Jan 4 17:02:10 mds1 kernel: BIOS-provided physical RAM map: Jan 4 17:02:10 mds1 kernel: BIOS-e820: 0000000000000000 - 000000000009dc00 (usable) Jan 4 17:02:10 mds1 kernel: BIOS-e820: 000000000009dc00 - 00000000000a0000 (reserved) Jan 4 17:02:10 mds1 kernel: BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved) Jan 4 17:02:10 mds1 kernel: BIOS-e820: 0000000000100000 - 000000003fffd000 (usable) Jan 4 17:02:10 mds1 kernel: BIOS-e820: 000000003fffd000 - 0000000040000000 (reserved) Jan 4 17:02:10 mds1 kernel: BIOS-e820: 00000000feffc000 - 00000000ff000000 (reserved) Jan 4 17:02:10 mds1 kernel: BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved) Jan 4 17:02:10 mds1 kernel: DMI 2.4 present. Jan 4 17:02:10 mds1 kernel: SMBIOS version 2.4 @ 0xFD760 Jan 4 17:02:10 mds1 kernel: last_pfn = 0x3fffd max_arch_pfn = 0x400000000 Jan 4 17:02:10 mds1 kernel: PAT not supported by CPU. Jan 4 17:02:10 mds1 kernel: init_memory_mapping: 0000000000000000-000000003fffd000 Jan 4 17:02:10 mds1 kernel: RAMDISK: 34a6c000 - 37fef97b Jan 4 17:02:10 mds1 kernel: ACPI: RSDP 00000000000fd730 00014 (v00 BOCHS ) Jan 4 17:02:10 mds1 kernel: ACPI: RSDT 000000003fffdc40 00034 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001) Jan 4 17:02:10 mds1 kernel: ACPI: FACP 000000003ffffe70 00074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001) Jan 4 17:02:10 mds1 kernel: ACPI: DSDT 000000003fffde40 01FB7 (v01 BXPC BXDSDT 00000001 INTL 20090123) Jan 4 17:02:10 mds1 kernel: ACPI: FACS 000000003ffffe00 00040 Jan 4 17:02:10 mds1 kernel: ACPI: SSDT 000000003fffdda0 0009E (v01 BOCHS BXPCSSDT 00000001 BXPC 00000001) Jan 4 17:02:10 mds1 kernel: ACPI: APIC 000000003fffdcc0 00072 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001) Jan 4 17:02:10 mds1 kernel: ACPI: HPET 000000003fffdc80 00038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001) Jan 4 17:02:10 mds1 kernel: No NUMA configuration found Jan 4 17:02:10 mds1 kernel: Faking a node at 0000000000000000-000000003fffd000 Jan 4 17:02:10 mds1 kernel: Bootmem setup node 0 0000000000000000-000000003fffd000 Jan 4 17:02:10 mds1 kernel: NODE_DATA [0000000000009000 - 000000000003cfff] Jan 4 17:02:10 mds1 kernel: bootmap [000000000003d000 - 0000000000044fff] pages 8 Jan 4 17:02:10 mds1 kernel: (7 early reservations) ==> bootmem [0000000000 - 003fffd000] Jan 4 17:02:10 mds1 kernel: #0 [0000000000 - 0000001000] BIOS data page ==> [0000000000 - 0000001000] Jan 4 17:02:10 mds1 kernel: #1 [0000006000 - 0000008000] TRAMPOLINE ==> [0000006000 - 0000008000] Jan 4 17:02:10 mds1 kernel: #2 [0001000000 - 0001f474e4] TEXT DATA BSS ==> [0001000000 - 0001f474e4] Jan 4 17:02:10 mds1 kernel: #3 [0034a6c000 - 0037fef97b] RAMDISK ==> [0034a6c000 - 0037fef97b] Jan 4 17:02:10 mds1 kernel: #4 [000009dc00 - 0000100000] BIOS reserved ==> [000009dc00 - 0000100000] Jan 4 17:02:10 mds1 kernel: #5 [0001f48000 - 0001f48079] BRK ==> [0001f48000 - 0001f48079] Jan 4 17:02:10 mds1 kernel: #6 [0000008000 - 0000009000] PGTABLE ==> [0000008000 - 0000009000] Jan 4 17:02:10 mds1 kernel: found SMP MP-table at [ffff8800000fd780] fd780 Jan 4 17:02:10 mds1 kernel: kvm-clock: Using msrs 12 and 11 Jan 4 17:02:10 mds1 kernel: kvm-clock: cpu 0, msr 0:1bbbf01, boot clock Jan 4 17:02:10 mds1 kernel: Zone PFN ranges: Jan 4 17:02:10 mds1 kernel: DMA 0x00000001 -> 0x00001000 Jan 4 17:02:10 mds1 kernel: DMA32 0x00001000 -> 0x00100000 Jan 4 17:02:10 mds1 kernel: Normal 0x00100000 -> 0x00100000 Jan 4 17:02:10 mds1 kernel: Movable zone start PFN for each node Jan 4 17:02:10 mds1 kernel: early_node_map[2] active PFN ranges Jan 4 17:02:10 mds1 kernel: 0: 0x00000001 -> 0x0000009d Jan 4 17:02:10 mds1 kernel: 0: 0x00000100 -> 0x0003fffd Jan 4 17:02:10 mds1 kernel: ACPI: PM-Timer IO Port: 0xb008 Jan 4 17:02:10 mds1 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled) Jan 4 17:02:10 mds1 kernel: ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) Jan 4 17:02:10 mds1 kernel: IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-23 Jan 4 17:02:10 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) Jan 4 17:02:10 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) Jan 4 17:02:10 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) Jan 4 17:02:10 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) Jan 4 17:02:10 mds1 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) Jan 4 17:02:10 mds1 kernel: Using ACPI (MADT) for SMP configuration information Jan 4 17:02:10 mds1 kernel: ACPI: HPET id: 0x8086a201 base: 0xfed00000 Jan 4 17:02:10 mds1 kernel: SMP: Allowing 1 CPUs, 0 hotplug CPUs Jan 4 17:02:10 mds1 kernel: PM: Registered nosave memory: 000000000009d000 - 000000000009e000 Jan 4 17:02:10 mds1 kernel: PM: Registered nosave memory: 000000000009e000 - 00000000000a0000 Jan 4 17:02:10 mds1 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000f0000 Jan 4 17:02:10 mds1 kernel: PM: Registered nosave memory: 00000000000f0000 - 0000000000100000 Jan 4 17:02:10 mds1 kernel: Allocating PCI resources starting at 40000000 (gap: 40000000:beffc000) Jan 4 17:02:10 mds1 kernel: Booting paravirtualized kernel on KVM Jan 4 17:02:10 mds1 kernel: NR_CPUS:4096 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1 Jan 4 17:02:10 mds1 kernel: PERCPU: Embedded 30 pages/cpu @ffff880002000000 s92504 r8192 d22184 u2097152 Jan 4 17:02:10 mds1 kernel: pcpu-alloc: s92504 r8192 d22184 u2097152 alloc=1*2097152 Jan 4 17:02:10 mds1 kernel: pcpu-alloc: [0] 0 Jan 4 17:02:10 mds1 kernel: kvm-clock: cpu 0, msr 0:2015f01, primary cpu clock Jan 4 17:02:10 mds1 kernel: Built 1 zonelists in Node order, mobility grouping on. Total pages: 258355 Jan 4 17:02:10 mds1 kernel: Policy zone: DMA32 Jan 4 17:02:10 mds1 kernel: Kernel command line: ro root=/dev/mapper/LustreVG-root rd_LVM_LV=LustreVG/root rd_LVM_LV=LustreVG/swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rhgb quiet Jan 4 17:02:10 mds1 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes) Jan 4 17:02:10 mds1 kernel: Checking aperture... Jan 4 17:02:10 mds1 kernel: No AGP bridge found Jan 4 17:02:10 mds1 kernel: AMD-Vi disabled by default: pass amd_iommu=on to enable Jan 4 17:02:10 mds1 kernel: Memory: 962756k/1048564k available (5016k kernel code, 400k absent, 85408k reserved, 6904k data, 1232k init) Jan 4 17:02:10 mds1 kernel: Hierarchical RCU implementation. Jan 4 17:02:10 mds1 kernel: NR_IRQS:33024 nr_irqs:256 Jan 4 17:02:10 mds1 kernel: Console: colour VGA+ 80x25 Jan 4 17:02:10 mds1 kernel: console [tty0] enabled Jan 4 17:02:10 mds1 kernel: allocated 10485760 bytes of page_cgroup Jan 4 17:02:10 mds1 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups Jan 4 17:02:10 mds1 kernel: HPET: 3 timers in total, 0 timers will be used for per-cpu timer Jan 4 17:02:10 mds1 kernel: Detected 1595.883 MHz processor. Jan 4 17:02:10 mds1 kernel: Calibrating delay loop (skipped) preset value.. 3191.76 BogoMIPS (lpj=1595883) Jan 4 17:02:10 mds1 kernel: pid_max: default: 32768 minimum: 301 Jan 4 17:02:10 mds1 kernel: Security Framework initialized Jan 4 17:02:10 mds1 kernel: SELinux: Initializing. Jan 4 17:02:10 mds1 kernel: Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Jan 4 17:02:10 mds1 kernel: Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) Jan 4 17:02:10 mds1 kernel: Mount-cache hash table entries: 256 Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys ns Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys cpuacct Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys memory Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys devices Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys freezer Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys net_cls Jan 4 17:02:10 mds1 kernel: Initializing cgroup subsys blkio Jan 4 17:02:10 mds1 kernel: mce: CPU supports 10 MCE banks Jan 4 17:02:10 mds1 kernel: alternatives: switching to unfair spinlock Jan 4 17:02:10 mds1 kernel: SMP alternatives: switching to UP code Jan 4 17:02:10 mds1 kernel: Freeing SMP alternatives: 33k freed Jan 4 17:02:10 mds1 kernel: ACPI: Core revision 20090903 Jan 4 17:02:10 mds1 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00 Jan 4 17:02:10 mds1 kernel: ftrace: allocating 20705 entries in 82 pages Jan 4 17:02:10 mds1 kernel: Setting APIC routing to flat Jan 4 17:02:10 mds1 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 Jan 4 17:02:10 mds1 kernel: CPU0: Intel QEMU Virtual CPU version 0.14.0 stepping 03 Jan 4 17:02:10 mds1 kernel: APIC calibration not consistent with PM-Timer: 102ms instead of 100ms Jan 4 17:02:10 mds1 kernel: APIC delta adjusted to PM-Timer: 6249876 (6375920) Jan 4 17:02:10 mds1 kernel: Performance Events: unsupported p6 CPU model 2 no PMU driver, software events only. Jan 4 17:02:10 mds1 kernel: NMI watchdog disabled (cpu0): hardware events not enabled Jan 4 17:02:10 mds1 kernel: Brought up 1 CPUs Jan 4 17:02:10 mds1 kernel: Total of 1 processors activated (3191.76 BogoMIPS). Jan 4 17:02:10 mds1 kernel: devtmpfs: initialized Jan 4 17:02:10 mds1 kernel: regulator: core version 0.5 Jan 4 17:02:10 mds1 kernel: NET: Registered protocol family 16 Jan 4 17:02:10 mds1 kernel: ACPI: bus type pci registered Jan 4 17:02:10 mds1 kernel: PCI: Using configuration type 1 for base access Jan 4 17:02:10 mds1 kernel: bio: create slab at 0 Jan 4 17:02:10 mds1 kernel: ACPI: Interpreter enabled Jan 4 17:02:10 mds1 kernel: ACPI: (supports S0 S3 S4 S5) Jan 4 17:02:10 mds1 kernel: ACPI: Using IOAPIC for interrupt routing Jan 4 17:02:10 mds1 kernel: ACPI: No dock devices found. Jan 4 17:02:10 mds1 kernel: ACPI: PCI Root Bridge [PCI0] (0000:00) Jan 4 17:02:10 mds1 kernel: pci 0000:00:01.3: quirk: region b000-b03f claimed by PIIX4 ACPI Jan 4 17:02:10 mds1 kernel: pci 0000:00:01.3: quirk: region b100-b10f claimed by PIIX4 SMB Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11) Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11) Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11) Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11) Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKS] (IRQs 9) *0 Jan 4 17:02:10 mds1 kernel: HEST: Table is not found! Jan 4 17:02:10 mds1 kernel: vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none Jan 4 17:02:10 mds1 kernel: vgaarb: loaded Jan 4 17:02:10 mds1 kernel: SCSI subsystem initialized Jan 4 17:02:10 mds1 kernel: usbcore: registered new interface driver usbfs Jan 4 17:02:10 mds1 kernel: usbcore: registered new interface driver hub Jan 4 17:02:10 mds1 kernel: usbcore: registered new device driver usb Jan 4 17:02:10 mds1 kernel: PCI: Using ACPI for IRQ routing Jan 4 17:02:10 mds1 kernel: NetLabel: Initializing Jan 4 17:02:10 mds1 kernel: NetLabel: domain hash size = 128 Jan 4 17:02:10 mds1 kernel: NetLabel: protocols = UNLABELED CIPSOv4 Jan 4 17:02:10 mds1 kernel: NetLabel: unlabeled traffic allowed by default Jan 4 17:02:10 mds1 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 Jan 4 17:02:10 mds1 kernel: hpet0: 3 comparators, 64-bit 100.000000 MHz counter Jan 4 17:02:10 mds1 kernel: Switching to clocksource kvm-clock Jan 4 17:02:10 mds1 kernel: pnp: PnP ACPI init Jan 4 17:02:10 mds1 kernel: ACPI: bus type pnp registered Jan 4 17:02:10 mds1 kernel: pnp: PnP ACPI: found 6 devices Jan 4 17:02:10 mds1 kernel: ACPI: ACPI bus type pnp unregistered Jan 4 17:02:10 mds1 kernel: NET: Registered protocol family 2 Jan 4 17:02:10 mds1 kernel: IP route cache hash table entries: 32768 (order: 6, 262144 bytes) Jan 4 17:02:10 mds1 kernel: TCP established hash table entries: 131072 (order: 9, 2097152 bytes) Jan 4 17:02:10 mds1 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) Jan 4 17:02:10 mds1 kernel: TCP: Hash tables configured (established 131072 bind 65536) Jan 4 17:02:10 mds1 kernel: TCP reno registered Jan 4 17:02:10 mds1 kernel: NET: Registered protocol family 1 Jan 4 17:02:10 mds1 kernel: pci 0000:00:00.0: Limiting direct PCI/PCI transfers Jan 4 17:02:10 mds1 kernel: pci 0000:00:01.0: PIIX3: Enabling Passive Release Jan 4 17:02:10 mds1 kernel: pci 0000:00:01.0: Activating ISA DMA hang workarounds Jan 4 17:02:10 mds1 kernel: Trying to unpack rootfs image as initramfs... Jan 4 17:02:10 mds1 kernel: Freeing initrd memory: 54798k freed Jan 4 17:02:10 mds1 kernel: audit: initializing netlink socket (disabled) Jan 4 17:02:10 mds1 kernel: type=2000 audit(1325714522.528:1): initialized Jan 4 17:02:10 mds1 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages Jan 4 17:02:10 mds1 kernel: VFS: Disk quotas dquot_6.5.2 Jan 4 17:02:10 mds1 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Jan 4 17:02:10 mds1 kernel: msgmni has been set to 1987 Jan 4 17:02:10 mds1 kernel: alg: No test for stdrng (krng) Jan 4 17:02:10 mds1 kernel: ksign: Installing public key data Jan 4 17:02:10 mds1 kernel: Loading keyring Jan 4 17:02:10 mds1 kernel: - Added public key CBA5A3F3E494381 Jan 4 17:02:10 mds1 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key) Jan 4 17:02:10 mds1 kernel: - Added public key D4A26C9CCD09BEDA Jan 4 17:02:10 mds1 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program Jan 4 17:02:10 mds1 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) Jan 4 17:02:10 mds1 kernel: io scheduler noop registered Jan 4 17:02:10 mds1 kernel: io scheduler anticipatory registered Jan 4 17:02:10 mds1 kernel: io scheduler deadline registered (default) Jan 4 17:02:10 mds1 kernel: io scheduler cfq registered Jan 4 17:02:10 mds1 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5 Jan 4 17:02:10 mds1 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4 Jan 4 17:02:10 mds1 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [1] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [2] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [3] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [4] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [5] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [6] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [7] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [8] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [9] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [10] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [11] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [12] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [13] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [14] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [15] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [16] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [17] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [18] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [19] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [20] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [21] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [22] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [23] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [24] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [25] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [26] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [27] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [28] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [29] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [30] registered Jan 4 17:02:10 mds1 kernel: acpiphp: Slot [31] registered Jan 4 17:02:10 mds1 kernel: pci-stub: invalid id string "" Jan 4 17:02:10 mds1 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 Jan 4 17:02:10 mds1 kernel: ACPI: Power Button [PWRF] Jan 4 17:02:10 mds1 kernel: processor LNXCPU:00: registered as cooling_device0 Jan 4 17:02:10 mds1 kernel: ERST: Table is not found! Jan 4 17:02:10 mds1 kernel: Non-volatile memory driver v1.3 Jan 4 17:02:10 mds1 kernel: Linux agpgart interface v0.103 Jan 4 17:02:10 mds1 kernel: crash memory driver: version 1.1 Jan 4 17:02:10 mds1 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled Jan 4 17:02:10 mds1 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A Jan 4 17:02:10 mds1 kernel: brd: module loaded Jan 4 17:02:10 mds1 kernel: loop: module loaded Jan 4 17:02:10 mds1 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1 Jan 4 17:02:10 mds1 kernel: Fixed MDIO Bus: probed Jan 4 17:02:10 mds1 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver Jan 4 17:02:10 mds1 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver Jan 4 17:02:10 mds1 kernel: uhci_hcd: USB Universal Host Controller Interface driver Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 Jan 4 17:02:10 mds1 kernel: uhci_hcd 0000:00:01.2: PCI INT D -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 Jan 4 17:02:10 mds1 kernel: uhci_hcd 0000:00:01.2: UHCI Host Controller Jan 4 17:02:10 mds1 kernel: uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1 Jan 4 17:02:10 mds1 kernel: uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c020 Jan 4 17:02:10 mds1 kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0001 Jan 4 17:02:10 mds1 kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 Jan 4 17:02:10 mds1 kernel: usb usb1: Product: UHCI Host Controller Jan 4 17:02:10 mds1 kernel: usb usb1: Manufacturer: Linux 2.6.32-131.6.1.el6_lustre.g65156ed.x86_64 uhci_hcd Jan 4 17:02:10 mds1 kernel: usb usb1: SerialNumber: 0000:00:01.2 Jan 4 17:02:10 mds1 kernel: usb usb1: configuration #1 chosen from 1 choice Jan 4 17:02:10 mds1 kernel: hub 1-0:1.0: USB hub found Jan 4 17:02:10 mds1 kernel: hub 1-0:1.0: 2 ports detected Jan 4 17:02:10 mds1 kernel: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 Jan 4 17:02:10 mds1 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1 Jan 4 17:02:10 mds1 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12 Jan 4 17:02:10 mds1 kernel: mice: PS/2 mouse device common for all mice Jan 4 17:02:10 mds1 kernel: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2 Jan 4 17:02:10 mds1 kernel: rtc_cmos 00:01: rtc core: registered rtc_cmos as rtc0 Jan 4 17:02:10 mds1 kernel: rtc0: alarms up to one day, 114 bytes nvram, hpet irqs Jan 4 17:02:10 mds1 kernel: cpuidle: using governor ladder Jan 4 17:02:10 mds1 kernel: cpuidle: using governor menu Jan 4 17:02:10 mds1 kernel: usbcore: registered new interface driver hiddev Jan 4 17:02:10 mds1 kernel: usbcore: registered new interface driver usbhid Jan 4 17:02:10 mds1 kernel: usbhid: v2.6:USB HID core driver Jan 4 17:02:10 mds1 kernel: TCP cubic registered Jan 4 17:02:10 mds1 kernel: Initializing XFRM netlink socket Jan 4 17:02:10 mds1 kernel: NET: Registered protocol family 17 Jan 4 17:02:10 mds1 kernel: registered taskstats version 1 Jan 4 17:02:10 mds1 kernel: rtc_cmos 00:01: setting system clock to 2012-01-04 22:02:01 UTC (1325714521) Jan 4 17:02:10 mds1 kernel: Initalizing network drop monitor service Jan 4 17:02:10 mds1 kernel: Freeing unused kernel memory: 1232k freed Jan 4 17:02:10 mds1 kernel: Write protecting the kernel read-only data: 10240k Jan 4 17:02:10 mds1 kernel: Freeing unused kernel memory: 1108k freed Jan 4 17:02:10 mds1 kernel: Freeing unused kernel memory: 1796k freed Jan 4 17:02:10 mds1 kernel: dracut: dracut-004-32.el6 Jan 4 17:02:10 mds1 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation Jan 4 17:02:10 mds1 kernel: device-mapper: uevent: version 1.0.3 Jan 4 17:02:10 mds1 kernel: device-mapper: ioctl: 4.20.6-ioctl (2011-02-02) initialised: dm-devel@redhat.com Jan 4 17:02:10 mds1 kernel: udev: starting version 147 Jan 4 17:02:10 mds1 kernel: dracut: Starting plymouth daemon Jan 4 17:02:10 mds1 kernel: dracut: rd_NO_DM: removing DM RAID activation Jan 4 17:02:10 mds1 kernel: dracut: rd_NO_MD: removing MD RAID activation Jan 4 17:02:10 mds1 kernel: scsi0 : ata_piix Jan 4 17:02:10 mds1 kernel: scsi1 : ata_piix Jan 4 17:02:10 mds1 kernel: ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc000 irq 14 Jan 4 17:02:10 mds1 kernel: ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc008 irq 15 Jan 4 17:02:10 mds1 kernel: input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3 Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10 Jan 4 17:02:10 mds1 kernel: virtio-pci 0000:00:03.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10 Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 10 Jan 4 17:02:10 mds1 kernel: virtio-pci 0000:00:06.0: PCI INT A -> Link[LNKB] -> GSI 10 (level, high) -> IRQ 10 Jan 4 17:02:10 mds1 kernel: virtio-pci 0000:00:07.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10 Jan 4 17:02:10 mds1 kernel: ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 Jan 4 17:02:10 mds1 kernel: sym53c8xx 0000:00:05.0: PCI INT A -> Link[LNKA] -> GSI 11 (level, high) -> IRQ 11 Jan 4 17:02:10 mds1 kernel: sym0: <895a> rev 0x0 at pci 0000:00:05.0 irq 11 Jan 4 17:02:10 mds1 kernel: sym0: No NVRAM, ID 7, Fast-40, LVD, parity checking Jan 4 17:02:10 mds1 kernel: sym0: SCSI BUS has been reset. Jan 4 17:02:10 mds1 kernel: scsi2 : sym-2.2.3 Jan 4 17:02:10 mds1 kernel: vda: vda1 vda2 Jan 4 17:02:10 mds1 kernel: scsi: waiting for bus probes to complete ... Jan 4 17:02:10 mds1 kernel: sym0: unknown interrupt(s) ignored, ISTAT=0x5 DSTAT=0x80 SIST=0x0 Jan 4 17:02:10 mds1 kernel: scsi 2:0:0:0: Direct-Access QEMU QEMU HARDDISK 0.14 PQ: 0 ANSI: 5 Jan 4 17:02:10 mds1 kernel: scsi target2:0:0: tagged command queuing enabled, command queue depth 16. Jan 4 17:02:10 mds1 kernel: scsi target2:0:0: Beginning Domain Validation Jan 4 17:02:10 mds1 kernel: scsi target2:0:0: Domain Validation skipping write tests Jan 4 17:02:10 mds1 kernel: scsi target2:0:0: Ending Domain Validation Jan 4 17:02:10 mds1 kernel: scsi 2:0:1:0: Direct-Access QEMU QEMU HARDDISK 0.14 PQ: 0 ANSI: 5 Jan 4 17:02:10 mds1 kernel: scsi target2:0:1: tagged command queuing enabled, command queue depth 16. Jan 4 17:02:10 mds1 kernel: scsi target2:0:1: Beginning Domain Validation Jan 4 17:02:10 mds1 kernel: scsi target2:0:1: Domain Validation skipping write tests Jan 4 17:02:10 mds1 kernel: scsi target2:0:1: Ending Domain Validation Jan 4 17:02:10 mds1 kernel: sd 2:0:0:0: [sda] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB) Jan 4 17:02:10 mds1 kernel: sd 2:0:0:0: [sda] Write Protect is off Jan 4 17:02:10 mds1 kernel: sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 4 17:02:10 mds1 kernel: sd 2:0:1:0: [sdb] 204800 512-byte logical blocks: (104 MB/100 MiB) Jan 4 17:02:10 mds1 kernel: sd 2:0:1:0: [sdb] Write Protect is off Jan 4 17:02:10 mds1 kernel: sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Jan 4 17:02:10 mds1 kernel: sda: Jan 4 17:02:10 mds1 kernel: sdb: unknown partition table Jan 4 17:02:10 mds1 kernel: sd 2:0:0:0: [sda] Attached SCSI disk Jan 4 17:02:10 mds1 kernel: unknown partition table Jan 4 17:02:10 mds1 kernel: sd 2:0:1:0: [sdb] Attached SCSI disk Jan 4 17:02:10 mds1 kernel: dracut: Scanning devices vda2 for LVM logical volumes LustreVG/root LustreVG/swap Jan 4 17:02:10 mds1 kernel: dracut: inactive '/dev/LustreVG/root' [1.37 GiB] inherit Jan 4 17:02:10 mds1 kernel: dracut: inactive '/dev/LustreVG/swap' [544.00 MiB] inherit Jan 4 17:02:10 mds1 kernel: EXT3-fs (dm-0): recovery required on readonly filesystem Jan 4 17:02:10 mds1 kernel: EXT3-fs (dm-0): write access will be enabled during recovery Jan 4 17:02:10 mds1 kernel: kjournald starting. Commit interval 5 seconds Jan 4 17:02:10 mds1 kernel: EXT3-fs (dm-0): recovery complete Jan 4 17:02:10 mds1 kernel: EXT3-fs (dm-0): mounted filesystem with ordered data mode Jan 4 17:02:10 mds1 kernel: dracut: Mounted root filesystem /dev/mapper/LustreVG-root Jan 4 17:02:10 mds1 kernel: dracut: Loading SELinux policy Jan 4 17:02:10 mds1 kernel: SELinux: Disabled at runtime. Jan 4 17:02:10 mds1 kernel: type=1404 audit(1325714526.046:2): selinux=0 auid=4294967295 ses=4294967295 Jan 4 17:02:10 mds1 kernel: dracut: /sbin/load_policy: Can't load policy: No such file or directory Jan 4 17:02:10 mds1 kernel: dracut: Switching root Jan 4 17:02:10 mds1 kernel: udev: starting version 147 Jan 4 17:02:10 mds1 kernel: piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0 Jan 4 17:02:10 mds1 kernel: Intel ICH 0000:00:04.0: PCI INT A -> Link[LNKD] -> GSI 11 (level, high) -> IRQ 11 Jan 4 17:02:10 mds1 kernel: intel8x0_measure_ac97_clock: measured 50970 usecs (4488 samples) Jan 4 17:02:10 mds1 kernel: intel8x0: measured clock 88051 rejected Jan 4 17:02:10 mds1 kernel: intel8x0_measure_ac97_clock: measured 50948 usecs (4484 samples) Jan 4 17:02:10 mds1 kernel: intel8x0: measured clock 88011 rejected Jan 4 17:02:10 mds1 kernel: intel8x0_measure_ac97_clock: measured 50949 usecs (4462 samples) Jan 4 17:02:10 mds1 kernel: intel8x0: measured clock 87577 rejected Jan 4 17:02:10 mds1 kernel: intel8x0: clocking to 48000 Jan 4 17:02:10 mds1 kernel: sd 2:0:0:0: Attached scsi generic sg0 type 0 Jan 4 17:02:10 mds1 kernel: sd 2:0:1:0: Attached scsi generic sg1 type 0 Jan 4 17:02:10 mds1 kernel: EXT3-fs (dm-0): using internal journal Jan 4 17:02:10 mds1 kernel: kjournald starting. Commit interval 5 seconds Jan 4 17:02:10 mds1 kernel: EXT3-fs (vda1): using internal journal Jan 4 17:02:10 mds1 kernel: EXT3-fs (vda1): mounted filesystem with ordered data mode Jan 4 17:02:10 mds1 kernel: Adding 557048k swap on /dev/mapper/LustreVG-swap. Priority:-1 extents:1 across:557048k Jan 4 17:02:10 mds1 kernel: microcode: CPU0 sig=0x623, pf=0x0, revision=0x0 Jan 4 17:02:10 mds1 kernel: platform microcode: firmware: requesting intel-ucode/06-02-03 Jan 4 17:02:10 mds1 kernel: Microcode Update Driver: v2.00 , Peter Oruba Jan 4 17:02:10 mds1 kernel: Microcode Update Driver: v2.00 removed. Jan 4 17:02:10 mds1 kernel: NET: Registered protocol family 10 Jan 4 17:02:10 mds1 kernel: lo: Disabled Privacy Extensions Jan 4 17:02:10 mds1 rpc.statd[1226]: Version 1.2.2 starting Jan 4 17:02:10 mds1 sm-notify[1227]: Version 1.2.2 starting Jan 4 17:02:10 mds1 kernel: RPC: Registered udp transport module. Jan 4 17:02:10 mds1 kernel: RPC: Registered tcp transport module. Jan 4 17:02:10 mds1 kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. Jan 4 17:02:10 mds1 avahi-daemon[1301]: Found user 'avahi' (UID 70) and group 'avahi' (GID 70). Jan 4 17:02:10 mds1 avahi-daemon[1301]: Successfully dropped root privileges. Jan 4 17:02:10 mds1 avahi-daemon[1301]: avahi-daemon 0.6.25 starting up. Jan 4 17:02:10 mds1 avahi-daemon[1301]: WARNING: No NSS support for mDNS detected, consider installing nss-mdns! Jan 4 17:02:10 mds1 avahi-daemon[1301]: Successfully called chroot(). Jan 4 17:02:10 mds1 avahi-daemon[1301]: Successfully dropped remaining capabilities. Jan 4 17:02:10 mds1 avahi-daemon[1301]: Loading service file /services/ssh.service. Jan 4 17:02:10 mds1 avahi-daemon[1301]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.122.89. Jan 4 17:02:10 mds1 avahi-daemon[1301]: New relevant interface eth0.IPv4 for mDNS. Jan 4 17:02:10 mds1 avahi-daemon[1301]: Network interface enumeration completed. Jan 4 17:02:10 mds1 avahi-daemon[1301]: Registering new address record for fe80::5054:ff:fed9:98ec on eth0.*. Jan 4 17:02:10 mds1 avahi-daemon[1301]: Registering new address record for 192.168.122.89 on eth0.IPv4. Jan 4 17:02:10 mds1 avahi-daemon[1301]: Registering HINFO record with values 'X86_64'/'LINUX'. Jan 4 17:02:10 mds1 kernel: TECH PREVIEW: fscache may not be fully supported. Jan 4 17:02:10 mds1 kernel: Please review provided documentation for limitations. Jan 4 17:02:10 mds1 kernel: Disabling lock debugging due to kernel taint Jan 4 17:02:10 mds1 kernel: Slow work thread pool: Starting up Jan 4 17:02:10 mds1 kernel: Slow work thread pool: Ready Jan 4 17:02:10 mds1 kernel: FS-Cache: Loaded Jan 4 17:02:10 mds1 kernel: FS-Cache: Netfs 'nfs' registered for caching Jan 4 17:02:10 mds1 acpid: starting up Jan 4 17:02:10 mds1 acpid: 1 rule loaded Jan 4 17:02:10 mds1 acpid: waiting for events: event logging is off Jan 4 17:02:11 mds1 acpid: client connected from 1385[68:68] Jan 4 17:02:11 mds1 acpid: 1 client rule loaded Jan 4 17:02:11 mds1 avahi-daemon[1301]: Server startup complete. Host name is mds1.local. Local service cookie is 3214687846. Jan 4 17:02:11 mds1 avahi-dnsconfd[1526]: Successfully connected to Avahi daemon. Jan 4 17:02:12 mds1 avahi-daemon[1301]: Service "mds1" (/services/ssh.service) successfully established.